Platform interaction interface self-adaptive adjustment method and system based on image recognition

Abstract

The invention relates to the technical field of adaptive interaction, and discloses a platform interaction interface adaptive adjustment method and system based on image recognition, and the method comprises the steps: obtaining a user form image; performing facial expression recognition according to the user form image to obtain a facial emotion component; performing body language recognition according to the user form image to obtain a body emotion component; performing calculation according to the facial emotion component and the limb emotion component to obtain a user emotion vector; adjusting the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy; and calculating an attention score according to the user emotion vector, and adjusting the teaching content difficulty of the platform according to the attention score and a preset teaching content difficulty automatic adjustment strategy. The method has the following effects that the platform interaction interface can be adaptively adjusted according to the use emotion of the user, and the adjustment efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of adaptive interaction technology, and in particular to a method and system for adaptively adjusting a platform interaction interface based on image recognition. Background Art

[0002] At present, with the continuous development of science and technology, various learning platforms are constantly emerging, providing people with convenient, flexible and diverse learning methods. Through the learning platform, people can learn knowledge anytime and anywhere, no longer restricted by traditional classrooms and time constraints. However, existing learning platforms often have the problem of relatively fixed interactive interface design, resulting in poor user experience. The design of the platform interactive interface is often fixed, and users are faced with a unified interface, which will make users feel bored and dull during long-term browsing and learning. Therefore, an adaptive adjustment method is needed to adjust the interface design according to the user's emotions and behaviors, so as to improve the user's learning experience.

[0003] In one existing technology, the following steps are included: detecting the current location information corresponding to the terminal device, and determining the interaction interface of the current platform as the target interaction interface when judging from the detection result that the current location information satisfies a first preset condition; automatically adjusting the content display orientation of the platform interaction interface according to the image of the user's eyes and the movement direction of the user's head; acquiring user behavior information, and replacing the display information in the target interaction interface with the target display information when judging from the acquisition result that the acquired user behavior information satisfies a second preset condition.

[0004] However, the prior art adopts a passive response mode, that is, it only makes adjustments when it detects changes in user behavior or emotions, rather than actively predicting and adapting. This approach results in low efficiency in adjusting the platform interface. Summary of the invention

[0005] The present invention provides a method and system for adaptively adjusting a platform interaction interface based on image recognition, so as to improve the adjustment efficiency of the platform interface.

[0006] In a first aspect, in order to solve the above technical problems, the present invention provides a platform interactive interface adaptive method based on image recognition, comprising: Get user form image; Perform facial expression recognition based on the user morphology image to obtain facial emotion components; Performing body language recognition based on the user morphology image to obtain a body language emotion component; Calculating according to the facial emotion component and the body emotion component to obtain a user emotion vector; Adjusting the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy; An attention score is calculated based on the user emotion vector, and the difficulty of the platform's teaching content is adjusted based on the attention score and a preset teaching content difficulty automatic adjustment strategy.

[0007] In an optional implementation, performing facial expression recognition according to the user morphological image to obtain facial emotion components includes: According to the user morphological image, a user facial image is obtained by using a face detection algorithm; Performing key point analysis on the user's facial image to obtain the mouth corner positioning point, facial center point, eye corner positioning point, face length and eyebrow positioning point; The mouth corner emotion component is obtained by calculating the angle between the first vector and the second vector, wherein the first vector is a vector formed by the left mouth corner positioning point and the facial center point, and the second vector is a vector formed by the right mouth corner positioning point and the facial center point; Wherein, the mouth corner positioning points include the left mouth corner positioning point and the right mouth corner positioning point; The eye corner emotion component is obtained by calculating the distance between the eye corner positioning point and the facial center point and dividing the distance by the face length; The eyebrow emotion component is calculated by the following formula: ; in, For the emotional weight of eyebrows, is the total number of eyebrow positioning points, is the number of the eyebrow positioning point, For the The vertical coordinate of the eyebrow positioning point, For the The vertical coordinate of the eyebrow positioning point; The facial emotion components include the eyebrow emotion component, the mouth corner emotion component and the eye corner emotion component.

[0008] In an optional implementation, the performing body language recognition according to the user form image to obtain the body language emotion component includes: According to the user morphological image, a user limb image is obtained by using a limb detection algorithm; Perform key point analysis based on the user's limb image to obtain facial orientation, neck positioning points, arm positioning points, and hip positioning points; The sitting posture emotion component is obtained by calculating the neck-hip distance between the neck positioning point and the hip positioning point and dividing the calculated distance by the pre-acquired height. taking the facial orientation as an orientation emotion component; The arm emotion component is calculated by the following formula: ; in, For the emotional weight of the arm, is the total number of arm positioning points, is the number of the arm positioning point, For the The vertical coordinate of the arm positioning point, For the The vertical coordinate of each arm positioning point; The body emotion component includes: the arm emotion component, the sitting posture emotion component and the orientation emotion component.

[0009] In an optional implementation, the calculating according to the facial emotion component and the body emotion component to obtain the user emotion vector includes: The user emotion vector is calculated by the following formula: ; in, , and Respectively represent the first dimension component, the second dimension component and the third dimension component of the user's emotion vector, is the weight of the eyebrow emotion component, is the weight of the emotion component of the corner of mouth, is the weight of the eye corner emotion component, is the weight of the arm's emotion component, is the weight of the sitting posture emotion component, is the weight towards the sentiment component, For the emotional weight of eyebrows, The emotional weight of the corners of the mouth. For the emotional weight of the corners of the eyes, For the emotional weight of the arm, For the emotional weight of sitting posture, Towards emotional weight.

[0010] In an optional implementation, the adjusting the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy includes: The user emotion vector includes a first dimension component, a second dimension component and a third dimension component; When the first dimensional component, the second dimensional component and the third dimensional component are all positive numbers, and the sum of the first dimensional component, the second dimensional component and the third dimensional component is less than a preset first threshold, the area ratio of the left column of the platform is expanded to a preset first area; reducing the content column area of ​​the platform to a preset second area; Reduce the right column area of ​​the platform to the preset third area.

[0011] In an optional implementation, the step of calculating the attention score according to the user emotion vector and adjusting the difficulty of the teaching content of the platform according to the attention score and a preset automatic adjustment strategy for the difficulty of the teaching content includes: The attention score is calculated by the following formula: ; in, Score for attention, , , are the first weight coefficient, the second weight coefficient, and the third weight coefficient respectively. , and Respectively represent the first dimension component, the second dimension component and the third dimension component of the user's emotion vector; When the attention score is less than a first attention threshold, adjusting the difficulty of the teaching content to simple; When the attention score is greater than the second attention threshold, the difficulty of the teaching content is adjusted to difficult.

[0012] In an optional implementation, after adjusting the difficulty of the teaching content to simple when the attention score is less than the first attention threshold, the method further includes: Get the duration of easy difficulty; When the duration of the simple difficulty is higher than a preset fatigue time threshold, a rest reminder is sent to the user.

[0013] In a second aspect, the present invention provides a platform interactive interface adaptive adjustment system based on image recognition, comprising: A data acquisition module, used to acquire a user morphological image; A facial recognition module, used to perform facial expression recognition based on the user morphological image to obtain facial emotion components; A body language recognition module, used to perform body language recognition based on the user morphology image to obtain a body language emotion component; An emotion quantification module, used to calculate according to the facial emotion component and the body emotion component to obtain a user emotion vector; A page layout module, used to adjust the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy; The content difficulty module is used to calculate the attention score according to the user emotion vector, and adjust the teaching content difficulty of the platform according to the attention score and a preset teaching content difficulty automatic adjustment strategy.

[0014] In a third aspect, the present invention also provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements any one of the above-mentioned methods for adaptively adjusting the platform interaction interface based on image recognition.

[0015] In a fourth aspect, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned methods for adaptive adjustment of the platform interaction interface based on image recognition.

[0016] Compared with the prior art, the present invention has the following beneficial effects: the present invention discloses a method and system for adaptively adjusting a platform interactive interface based on image recognition, the method comprising obtaining a user morphological image; performing facial expression recognition according to the user morphological image to obtain a facial emotion component; performing body language recognition according to the user morphological image to obtain a body emotion component; performing calculations according to the facial emotion component and the body emotion component to obtain a user emotion vector; adjusting the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy; calculating an attention score according to the user emotion vector, and adjusting the teaching content difficulty of the platform according to the attention score and a preset teaching content difficulty automatic adjustment strategy. Different from the passive response mode adopted by the prior art, the present invention can actively predict the user's emotional state by real-time collection and analysis of the user's facial expression and body language, and dynamically adjust the platform's page layout and teaching content difficulty accordingly, thereby improving the adjustment efficiency of the platform interface.

[0017] Specifically, the present invention not only pays attention to the user's facial expressions, but also takes into account body language, and through comprehensive analysis of facial emotion components and body emotion components, the user's emotional state is more comprehensively understood. Specifically, it includes obtaining eyebrow emotion components, mouth corner emotion components and eye corner emotion components through facial expression recognition, and obtaining arm emotion components, sitting posture emotion components and orientation emotion components through body language recognition. At the same time, the present invention proposes a user emotion quantification formula, which converts the emotional state into quantifiable data by calculating the user's emotion vector, and then automatically adjusts the page layout and teaching content difficulty according to these data. Furthermore, the present invention introduces an attention evaluation mechanism, and by calculating the attention score, when the user's attention score is lower than a certain threshold, the difficulty of the teaching content will be automatically adjusted to simple, and when the duration of the simple difficulty exceeds the preset fatigue time threshold, a rest reminder will be sent to the user, which effectively prevents the user from being overly fatigued and ensures the learning effect. In summary, the present invention realizes the adaptive adjustment of the platform interaction interface based on image recognition, and improves the adjustment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic flow chart of a method for adaptively adjusting a platform interaction interface based on image recognition provided by a first embodiment of the present invention; Figure 2 It is a schematic diagram of the structure of a platform interactive interface adaptive adjustment system based on image recognition provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] At present, with the continuous development of science and technology, various learning platforms are constantly emerging, providing people with convenient, flexible and diverse learning methods. Through the learning platform, people can learn knowledge anytime and anywhere, no longer restricted by traditional classrooms and time constraints. However, existing learning platforms often have the problem of relatively fixed interactive interface design, resulting in poor user experience. The design of the platform interactive interface is often fixed, and users are faced with a unified interface, which will make users feel bored and dull during long-term browsing and learning. Therefore, an adaptive adjustment method is needed to adjust the interface design according to the user's emotions and behaviors, so as to improve the user's learning experience.

[0021] In one existing technology, the following steps are included: detecting the current location information corresponding to the terminal device, and determining the interaction interface of the current platform as the target interaction interface when judging from the detection result that the current location information satisfies a first preset condition; automatically adjusting the content display orientation of the platform interaction interface according to the image of the user's eyes and the movement direction of the user's head; acquiring user behavior information, and replacing the display information in the target interaction interface with the target display information when judging from the acquisition result that the acquired user behavior information satisfies a second preset condition.

[0022] However, the prior art adopts a passive response mode, that is, it only makes adjustments when it detects changes in user behavior or emotions, rather than actively predicting and adapting. This approach results in low efficiency in adjusting the platform interface.

[0023] To solve the above problems, refer to Figure 1 The first embodiment of the present invention provides a method for adaptively adjusting a platform interaction interface based on image recognition, comprising the following steps: S11, obtaining a user form image; S12, performing facial expression recognition according to the user morphological image to obtain facial emotion components; S13, performing body language recognition according to the user morphology image to obtain a body language emotion component; S14, calculating according to the facial emotion component and the body emotion component to obtain a user emotion vector; S15, adjusting the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy; S16, calculating an attention score according to the user emotion vector, and adjusting the difficulty of the platform's teaching content according to the attention score and a preset teaching content difficulty automatic adjustment strategy.

[0024] In step S11, a user form image is acquired.

[0025] In one embodiment, the platform uses a high-resolution camera to collect images of children. The camera can be built into the platform's device, such as a tablet computer, a smart TV, or a dedicated educational robot. The platform can also be connected to an external camera to provide higher image quality and a wider viewing angle. Before collecting the image, the platform will prompt the child to maintain a specific posture, such as facing the camera head-on, through voice or text. The collected image is then histogram-equalized to improve the image's lighting conditions and increase the image's contrast and brightness. The user morphology image is obtained for subsequent image processing operations.

[0026] In step S12, facial expression recognition is performed based on the user form image to obtain facial emotion components.

[0027] In one implementation, a user facial image is obtained based on the user morphological image by using a face detection algorithm; Performing key point analysis on the user's facial image to obtain the mouth corner positioning point, facial center point, eye corner positioning point, face length and eyebrow positioning point; The mouth corner emotion component is obtained by calculating the angle between the first vector and the second vector, wherein the first vector is a vector formed by the left mouth corner positioning point and the facial center point, and the second vector is a vector formed by the right mouth corner positioning point and the facial center point; Wherein, the mouth corner positioning points include the left mouth corner positioning point and the right mouth corner positioning point; The eye corner emotion component is obtained by calculating the distance between the eye corner positioning point and the facial center point and dividing the distance by the face length; The eyebrow emotion component is calculated by the following formula: ; in, For the emotional weight of eyebrows, is the total number of eyebrow positioning points, is the number of the eyebrow positioning point, For the The vertical coordinate of the eyebrow positioning point, For the The vertical coordinate of the eyebrow positioning point; The facial emotion components include the eyebrow emotion component, the mouth corner emotion component and the eye corner emotion component.

[0028] It is worth mentioning that the face detection algorithm can use the YOLO algorithm to input the user's morphological image into the model. The model outputs a detection frame, each of which contains the location information of the face. The high-confidence face detection frame is selected based on the confidence threshold. The Facial Landmark Detection model is then used for key point detection. This model can accurately detect key points on the face, such as eyes, nose, mouth, eyebrows, etc. Including: inputting the facial image cropped from the face detection frame into the model. The model outputs the location of key points on the face, including the corner of the mouth positioning point, the center of the face, the corner of the eye positioning point, the eyebrow positioning point, etc.

[0029] It is worth mentioning that YOLO is a real-time target detection system that contains 24 convolutional layers and 2 fully connected layers. The convolutional layer is used to extract image features, and the fully connected layer is used to predict the bounding box and category probability of the target. YOLO divides the input image into S×S grids. Each grid unit is responsible for detecting targets whose center points fall within the grid. Each grid unit predicts B bounding boxes and the confidence of the bounding box. The confidence includes two aspects: one is the probability that the bounding box contains the target, and the other is the accuracy of the bounding box. The accuracy of the bounding box is characterized by the intersection of the predicted box and the actual box. The YOLO model is trained with face images from big data. This method completes the detection of faces by directly calling the trained YOLO model.

[0030] It is worth mentioning that facial landmark detection refers to locating a series of points with specific semantic information on a 2D face image, such as eyes, nose, mouth, etc. DTLD is a facial landmark detection model that combines convolutional neural network (CNN) and Transformer. Similarly, using face images in big data for training, this method directly calls the DTLD model to locate the key points of the face.

[0031] It is worth noting that the larger the value of the eyebrow emotion component, the more surprised or curious the child is about the current learning content. The corner of mouth emotion component can be divided into corner of mouth upturned and corner of mouth downturned, which is expressed by the positive and negative values ​​of the corner of mouth emotion component. When the corner of mouth upturned, the larger the value of the corner of mouth emotion component, the more happy or satisfied the child is. Downturned corner of mouth indicates that the child feels sad or disappointed. The larger the value of the corner of eye emotion component, the happier the child is.

[0032] In step S13, body language recognition is performed based on the user form image to obtain a body language emotion component.

[0033] In one implementation, according to the user morphological image, a user limb image is obtained by using a limb detection algorithm; Perform key point analysis based on the user's limb image to obtain facial orientation, neck positioning points, arm positioning points, and hip positioning points; The sitting posture emotion component is obtained by calculating the neck-hip distance between the neck positioning point and the hip positioning point and dividing the calculated distance by the pre-acquired height. taking the facial orientation as an orientation emotion component; The arm emotion component is calculated by the following formula: ; in, For the emotional weight of the arm, is the total number of arm positioning points, is the number of the arm positioning point, For the The vertical coordinate of the arm positioning point, For the The vertical coordinate of each arm positioning point; The body emotion component includes: the arm emotion component, the sitting posture emotion component and the orientation emotion component.

[0034] It is worth noting that the MediaPipe Holistic model is a comprehensive model provided by MediaPipe that can simultaneously detect the key points of the face, hands, and posture of the human body. The MediaPipe Holistic model has been trained with historical limb big data. This method calls the API interface to pass the user's image or video frame through the model to complete the extraction of limb key points and the positioning of key points, including facial orientation, neck positioning points, arm positioning points, and hip positioning points.

[0035] It is worth noting that the sitting posture emotion component can reflect a person's comfort and concentration. It means that the child sits more upright and feels confident, focused or excited. If the sitting posture emotion component is small, the child is in a negative emotion. The platform adjusts the teaching content and adds interactive links to improve the child's attention and interest. For the orientation emotion component, the facial orientation can be divided into front, side, back and other directions. Front orientation is associated with positive social interaction and concentration, while side or back orientation is associated with distraction or disinterest. For the arm emotion component, the movement of the arm can reflect the child's activity and emotional state. Active arm movements are associated with positive emotional states, while static or stiff arm movements are associated with negative emotional states. A larger value means that the child's arm movements are more active, and he or she feels excited, curious or highly involved. If the arm emotion component is small, the platform will consider adding interactive links or adjusting the teaching content to improve the child's participation and interest.

[0036] In step S14, calculation is performed based on the facial emotion component and the body emotion component to obtain a user emotion vector.

[0037] In one implementation, the user emotion vector is calculated using the following formula: ; in, , and Respectively represent the first dimension component, the second dimension component and the third dimension component of the user's emotion vector, is the weight of the eyebrow emotion component, is the weight of the emotion component of the corner of mouth, is the weight of the eye corner emotion component, is the weight of the arm's emotion component, is the weight of the sitting posture emotion component, is the weight towards the sentiment component, For the emotional weight of eyebrows, The emotional weight of the corners of the mouth. For the emotional weight of the corners of the eyes, For the emotional weight of the arm, For the emotional weight of sitting posture, Towards emotional weight.

[0038] It is worth noting that the first dimension component reflects the comprehensive emotional state of children's facial expressions and arm movements, the second dimension component reflects the comprehensive emotional state of children's facial expressions and orientation, and the third dimension component reflects the comprehensive emotional state of children's facial expressions, sitting posture and orientation. The weights are set to adjust the importance of facial expressions and body movements. When facial expressions are more important in emotion recognition, their weights can be increased. The higher the value of each component, the more positive the user's emotions are. At this time, the platform can continue to provide similar experimental activities to maintain children's interest and enthusiasm. When the value of each component is low, it means that the child feels tired or is not interested in the current mathematical content. The platform needs to adjust the teaching content and increase interactive links to improve children's attention and interest.

[0039] In step S15, the page layout of the platform is adjusted according to the user emotion vector and a preset page layout automatic adjustment strategy.

[0040] In one embodiment, the user emotion vector includes a first dimensional component, a second dimensional component and a third dimensional component; when the first dimensional component, the second dimensional component and the third dimensional component are all positive numbers, and the sum of the first dimensional component, the second dimensional component and the third dimensional component is less than a preset first threshold, the area ratio of the left column of the platform is expanded to a preset first area; the area of ​​the content column of the platform is reduced to a preset second area; and the area of ​​the right column of the platform is reduced to a preset third area.

[0041] It is worth noting that the sum of the components of these three dimensions needs to be less than a preset first threshold. At this time, the user's learning mood is in a stable state and is not overly enthusiastic. Adjust the area ratio of the left column of the platform to the preset first area. This is because it is beneficial to increase the visibility of navigation or auxiliary information when the user's emotions are positive but not too strong. The first area is 1.1 times the original area. Adjust the content column area of ​​the platform to the preset second area. This is to reduce information overload while maintaining content accessibility. The second area is 0.9 times the original area. Adjust the right column area of ​​the platform to the preset third area. This is to balance the page layout and ensure that the user interface is neither too crowded nor too empty. The third area is 0.9 times the original area.

[0042] In step S16, an attention score is calculated based on the user emotion vector, and the difficulty of the platform's teaching content is adjusted based on the attention score and a preset teaching content difficulty automatic adjustment strategy.

[0043] In one implementation, the attention score is calculated using the following formula: ; in, Score for attention, , , are the first weight coefficient, the second weight coefficient, and the third weight coefficient respectively. , and Respectively represent the first dimension component, the second dimension component and the third dimension component of the user's emotion vector; When the attention score is less than the first attention threshold, the difficulty of the teaching content is adjusted to simple; when the attention score is greater than the second attention threshold, the difficulty of the teaching content is adjusted to difficult. The duration of the simple difficulty is obtained; when the duration of the simple difficulty is higher than the preset fatigue time threshold, a rest reminder is sent to the user.

[0044] It is worth noting that in order to ensure that the emotional components of different dimensions are comparable when calculating the attention score, the user's emotional vector needs to be normalized. The normalization method is to scale the value of each component to the range of [0, 1] by dividing it by the difference between the maximum and minimum values. The first attention threshold represents the lowest threshold of the user's learning state, for example, it is set to 0.4. When the attention score is lower than the first attention threshold, it means that the user's attention is not focused enough at this time and can only receive simple teaching content. The second attention threshold indicates that the user is highly focused, and the user can receive difficult content at this time, for example, it is set to 0.9. This method is not limited to this.

[0045] It is worth noting that the present invention is based on a children's education intelligent platform and belongs to the data processing link. The method analyzes data to achieve adaptive adjustment of the platform interface. The children's education intelligent platform is a comprehensive system that integrates multiple technologies and aims to improve the efficiency and quality of children's education. The data input link of the platform accurately captures the user's scanning behavior through the user intention recognition technology of the scanning pen, reduces misoperation, and obtains accurate input data. The data processing link includes an intelligent recommendation system for children's education content based on knowledge graphs, which analyzes children's learning data and preferences and recommends personalized education content for children; and an adaptive adjustment system for the platform interaction interface based on image recognition, which identifies the environment and equipment when children interact with the platform, automatically adjusts the interaction interface, and improves the user experience. In addition, there is an intelligent education platform interaction system based on children's voice behavior analysis, which understands children's language habits and expressions and provides a natural and friendly interactive experience. The data output link analyzes children's learning performance and progress through a children's mathematical thinking training effect evaluation system based on data mining, evaluates the effect of the education platform, and helps educators and parents adjust education strategies in a timely manner. Overall, the platform covers the entire process from data input to processing to output, providing children with a personalized, interactive and measurable educational environment.

[0046] In summary, the present invention proposes an adaptive adjustment method for a platform interaction interface based on image recognition, which aims to improve user experience and enhance interactivity by analyzing non-verbal signals such as user expressions and gestures. The method first collects user image information through a camera or other device, then extracts key features such as facial expressions, head angles, gestures, etc. from the image, and uses a pre-trained emotion recognition model to analyze the user's current emotional state. The present invention proposes a user emotion quantification formula, which converts the emotional state into quantifiable data by calculating the user's emotional vector, and then automatically adjusts the page layout and teaching content difficulty according to these data. According to the user's emotions and behavior patterns, the system dynamically adjusts the color, layout, font size and other elements of the user interface, such as simplifying the interface and reducing interference information when the user shows fatigue or confusion; providing more functions and detailed information when the user is active and focused; adjusting the interface color according to the user's emotional state, such as using soft tones when the user feels anxious; adjusting the position of the interface elements according to the user's line of sight and gestures, so that it is more in line with the user's operating habits. The present invention can adaptively adjust the platform interaction interface according to the user's usage emotions and improve the adjustment efficiency.

[0047] Reference Figure 2 The second embodiment of the present invention provides a platform interactive interface adaptive adjustment system based on image recognition, including: A data acquisition module, used to acquire a user morphological image; A facial recognition module, used for performing facial expression recognition based on the user morphological image to obtain facial emotion components; A body language recognition module, used to perform body language recognition based on the user morphology image to obtain a body language emotion component; An emotion quantification module, used to calculate according to the facial emotion component and the body emotion component to obtain a user emotion vector; A page layout module, used to adjust the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy; The content difficulty module is used to calculate the attention score according to the user emotion vector, and adjust the teaching content difficulty of the platform according to the attention score and a preset teaching content difficulty automatic adjustment strategy.

[0048] Preferably, the data acquisition module is used to: Get the user form image.

[0049] Preferably, the facial recognition module is used to: Performing facial expression recognition according to the user morphology image to obtain facial emotion components includes: According to the user morphological image, a user facial image is obtained by using a face detection algorithm; Performing key point analysis on the user's facial image to obtain the mouth corner positioning point, facial center point, eye corner positioning point, face length and eyebrow positioning point; The mouth corner emotion component is obtained by calculating the angle between the first vector and the second vector, wherein the first vector is a vector formed by the left mouth corner positioning point and the facial center point, and the second vector is a vector formed by the right mouth corner positioning point and the facial center point; Wherein, the mouth corner positioning points include the left mouth corner positioning point and the right mouth corner positioning point; The eye corner emotion component is obtained by calculating the distance between the eye corner positioning point and the facial center point and dividing the distance by the face length; The eyebrow emotion component is calculated by the following formula: ; in, For the emotional weight of eyebrows, is the total number of eyebrow positioning points, is the number of the eyebrow positioning point, For the The vertical coordinate of the eyebrow positioning point, For the The vertical coordinate of the eyebrow positioning point; The facial emotion components include the eyebrow emotion component, the mouth corner emotion component and the eye corner emotion component.

[0050] Preferably, the body recognition module is used to: Performing body language recognition according to the user form image to obtain body language emotion components includes: According to the user morphological image, a user limb image is obtained by using a limb detection algorithm; Perform key point analysis based on the user's limb image to obtain facial orientation, neck positioning points, arm positioning points, and hip positioning points; The sitting posture emotion component is obtained by calculating the neck-hip distance between the neck positioning point and the hip positioning point and dividing the calculated distance by the pre-acquired height. taking the facial orientation as an orientation emotion component; The arm emotion component is calculated by the following formula: ; in, For the emotional weight of the arm, is the total number of arm positioning points, is the number of the arm positioning point, For the The vertical coordinate of the arm positioning point, For the The vertical coordinate of each arm positioning point; The body emotion component includes: the arm emotion component, the sitting posture emotion component and the orientation emotion component.

[0051] Preferably, the emotion quantification module is used to calculate according to the facial emotion component and the body emotion component to obtain a user emotion vector, including: The user emotion vector is calculated by the following formula: ; in, , and Respectively represent the first dimension component, the second dimension component and the third dimension component of the user's emotion vector, is the weight of the eyebrow emotion component, is the weight of the emotion component of the corner of mouth, is the weight of the eye corner emotion component, is the weight of the arm's emotion component, is the weight of the sitting posture emotion component, is the weight towards the sentiment component, For the emotional weight of eyebrows, The emotional weight of the corners of the mouth. For the emotional weight of the corners of the eyes, For the emotional weight of the arm, For the emotional weight of sitting posture, Towards emotional weight.

[0052] Preferably, the page layout module is used to: The page layout of the platform is adjusted according to the user emotion vector and a preset page layout automatic adjustment strategy, including: The user emotion vector includes a first dimension component, a second dimension component and a third dimension component; When the first dimensional component, the second dimensional component and the third dimensional component are all positive numbers, and the sum of the first dimensional component, the second dimensional component and the third dimensional component is less than a preset first threshold, the area ratio of the left column of the platform is expanded to a preset first area; reducing the content column area of ​​the platform to a preset second area; Reduce the right column area of ​​the platform to the preset third area.

[0053] Preferably, the content difficulty module is used to: An attention score is calculated according to the user emotion vector, and the difficulty of the teaching content of the platform is adjusted according to the attention score and a preset teaching content difficulty automatic adjustment strategy, including: The attention score is calculated by the following formula: ; in, Score for attention, , , are the first weight coefficient, the second weight coefficient, and the third weight coefficient respectively. , and Respectively represent the first dimension component, the second dimension component and the third dimension component of the user's emotion vector; When the attention score is less than a first attention threshold, adjusting the difficulty of the teaching content to simple; When the attention score is greater than the second attention threshold, the difficulty of the teaching content is adjusted to difficult.

[0054] Preferably, after adjusting the difficulty of the teaching content to simple when the attention score is less than the first attention threshold, the method further includes: Get the duration of easy difficulty; When the duration of the simple difficulty is higher than a preset fatigue time threshold, a rest reminder is sent to the user.

[0055] It should be noted that the platform interaction interface adaptive adjustment system based on image recognition provided in an embodiment of the present invention is used to execute all the process steps of the platform interaction interface adaptive adjustment method based on image recognition in the above-mentioned embodiment. The working principles and beneficial effects of the two correspond one to one, and therefore will not be repeated here.

[0056] The embodiment of the present invention further provides an electronic device. The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a data acquisition program. When the processor executes the computer program, the steps in the above-mentioned embodiments of the method for adaptively adjusting the platform interaction interface based on image recognition are implemented, such as Figure 1 Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are realized, such as the data acquisition module.

[0057] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device.

[0058] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, and a smart tablet. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than the above components, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0059] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0060] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the electronic device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0061] Wherein, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0062] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0063] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for adaptively adjusting a platform interaction interface based on image recognition, characterized in that: include: Get user form image; Perform facial expression recognition based on the user morphology image to obtain facial emotion components; Performing body language recognition based on the user morphology image to obtain a body language emotion component; Calculating according to the facial emotion component and the body emotion component to obtain a user emotion vector; Adjusting the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy; An attention score is calculated based on the user emotion vector, and the difficulty of the platform's teaching content is adjusted based on the attention score and a preset teaching content difficulty automatic adjustment strategy.

2. The method for adaptively adjusting the platform interaction interface based on image recognition according to claim 1, characterized in that: The step of performing facial expression recognition according to the user morphological image to obtain facial emotion components includes: According to the user morphological image, a user facial image is obtained by using a face detection algorithm; Performing key point analysis on the user's facial image to obtain the mouth corner positioning point, facial center point, eye corner positioning point, face length and eyebrow positioning point; The mouth corner emotion component is obtained by calculating the angle between the first vector and the second vector, wherein the first vector is a vector formed by the left mouth corner positioning point and the facial center point, and the second vector is a vector formed by the right mouth corner positioning point and the facial center point; Wherein, the mouth corner positioning points include the left mouth corner positioning point and the right mouth corner positioning point; The eye corner emotion component is obtained by calculating the distance between the eye corner positioning point and the facial center point and dividing the distance by the face length; The eyebrow emotion component is calculated by the following formula: ； in, For the emotional weight of eyebrows, is the total number of eyebrow positioning points, is the number of the eyebrow positioning point, For the The vertical coordinate of the eyebrow positioning point, For the The vertical coordinate of the eyebrow positioning point; The facial emotion components include the eyebrow emotion component, the mouth corner emotion component and the eye corner emotion component.

3. The method for adaptively adjusting the platform interaction interface based on image recognition according to claim 1, characterized in that: The step of performing body language recognition according to the user form image to obtain a body language emotion component includes: According to the user morphological image, a user limb image is obtained by using a limb detection algorithm; Perform key point analysis based on the user's limb image to obtain facial orientation, neck positioning points, arm positioning points, and hip positioning points; The sitting posture emotion component is obtained by calculating the neck-hip distance between the neck positioning point and the hip positioning point and dividing the calculated distance by the pre-acquired height. taking the facial orientation as an orientation emotion component; The arm emotion component is calculated by the following formula: ； in, For the emotional weight of the arm, is the total number of arm positioning points, is the number of the arm positioning point, For the The vertical coordinate of the arm positioning point, For the The vertical coordinate of the arm positioning point; The body emotion component includes: the arm emotion component, the sitting posture emotion component and the orientation emotion component.

4. The method for adaptively adjusting the platform interaction interface based on image recognition according to claim 1, characterized in that: The step of calculating according to the facial emotion component and the body emotion component to obtain a user emotion vector includes: The user emotion vector is calculated by the following formula: ； in, , and Respectively represent the first dimension component, the second dimension component and the third dimension component of the user's emotion vector, is the weight of the eyebrow emotion component, is the weight of the emotion component of the corner of mouth, is the weight of the eye corner emotion component, is the weight of the arm's emotion component, is the weight of the sitting posture emotion component, is the weight towards the sentiment component, For the emotional weight of eyebrows, The emotional weight of the corners of the mouth. For the emotional weight of the corners of the eyes, For the emotional weight of the arm, For the emotional weight of sitting posture, Towards emotional weight.

5. The method for adaptively adjusting the platform interaction interface based on image recognition according to claim 1, characterized in that: The step of adjusting the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy includes: The user emotion vector includes a first dimension component, a second dimension component and a third dimension component; When the first dimensional component, the second dimensional component and the third dimensional component are all positive numbers, and the sum of the first dimensional component, the second dimensional component and the third dimensional component is less than a preset first threshold, the area ratio of the left column of the platform is expanded to a preset first area; reducing the content column area of ​​the platform to a preset second area; Reduce the right column area of ​​the platform to the preset third area.

6. The method for adaptively adjusting the platform interaction interface based on image recognition according to claim 1, characterized in that: The calculating of the attention score according to the user emotion vector and adjusting the difficulty of the teaching content of the platform according to the attention score and a preset automatic adjustment strategy of the difficulty of the teaching content include: The attention score is calculated by the following formula: ； in, Score for attention, , , are the first weight coefficient, the second weight coefficient, and the third weight coefficient respectively. , and Respectively represent the first dimension component, the second dimension component and the third dimension component of the user's emotion vector; When the attention score is less than a first attention threshold, adjusting the difficulty of the teaching content to simple; When the attention score is greater than the second attention threshold, the difficulty of the teaching content is adjusted to difficult.

7. The method for adaptively adjusting the platform interaction interface based on image recognition according to claim 6, characterized in that: After adjusting the difficulty of the teaching content to simple when the attention score is less than the first attention threshold, the method further includes: Get the duration of easy difficulty; When the duration of the simple difficulty is higher than a preset fatigue time threshold, a rest reminder is sent to the user.

8. A platform interactive interface adaptive adjustment system based on image recognition, characterized in that: include: A data acquisition module, used to acquire a user morphological image; A facial recognition module, used to perform facial expression recognition based on the user morphological image to obtain facial emotion components; A body language recognition module, used to perform body language recognition based on the user morphology image to obtain a body language emotion component; An emotion quantification module, used to calculate according to the facial emotion component and the body emotion component to obtain a user emotion vector; A page layout module, used to adjust the page layout of the platform according to the user emotion vector and a preset page layout automatic adjustment strategy; Content Difficulty Module for An attention score is calculated based on the user emotion vector, and the difficulty of the platform's teaching content is adjusted based on the attention score and a preset teaching content difficulty automatic adjustment strategy.

9. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the platform interaction interface adaptive adjustment method based on image recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the platform interaction interface adaptive adjustment method based on image recognition as described in any one of claims 1 to 7.

Images