Display data integrated processing system and method based on multi-sensor fusion

Abstract

The invention discloses a display data integration processing system and method based on multi-sensor fusion, and relates to the technical field of display data integration processing, the system comprises a sensor data acquisition module, a mobile display data integration module, an instruction processing module and a display module; the output end of the sensor data acquisition module is connected with the input end of the mobile display data integration module; the output end of the mobile display data integration module is connected with the input end of the instruction processing module; the output end of the instruction processing module is connected with the input end of the display module. The visual problem of the mobile display in a fitness learning scene is mainly solved, intelligent instruction analysis is formed based on integrated processing of sensor data, and the further intelligent level of the mobile display is promoted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of display data integration processing, in particular to a display data integration processing system and method based on multi-sensor fusion. BACKGROUND

[0002] Mobile display, also known as mobile smart screen, is also called "buddy machine" on the network. It is composed of a screen and a movable floor stand. As a multifunctional product integrating TV, tablet and sound in recent years, mobile display is mainly used in home scenarios to realize course learning, video fitness and other functions. However, in the current technical development, mobile display still has certain intelligent defects, especially in the main fitness learning scenario. Due to different movements of fitness content, users often need to turn their bodies or face away from the mobile display in certain movements, which causes the user to be unable to watch the subsequent movements of the fitness video in time, causing certain disturbance. At the same time, different users have different levels of mastery of fitness movements. Therefore, there is currently a lack of mobile display that can make intelligent feedback according to the user's own behavior. SUMMARY

[0003] The purpose of the present application is to provide a display data integration processing system and method based on multi-sensor fusion to solve the problems in the prior art.

[0004] To achieve the above purpose, the present application provides the following technical scheme: a display data integration processing method based on multi-sensor fusion, the method comprising the following steps:

[0005] S1. A plurality of sensors are arranged around the mobile display to identify user behavior movements and mobile display peripheral environment data and feed back to the mobile display data integration platform;

[0006] S2. The mobile display data integration platform integrates, analyzes and processes the data to form a mobile start instruction and a mobile termination command;

[0007] S3. Based on the mobile start instruction and the mobile display peripheral environment data, the mobile display is controlled to move intelligently and stop moving when the mobile termination command is reached;

[0008] S4. Identify user behavior movements to control the mobile display to continue video display.

[0009] According to the above technical scheme, the identification of user behavior movements and mobile display peripheral environment data comprises:

[0010] Based on the plurality of sensors arranged around the mobile display, the center point of the mobile rod of the mobile display is taken as the center of a right-angle coordinate system in the space of the region formed in the horizontal and vertical directions, the objects in the region are monitored, and the object coordinates are marked.

[0011] The user and the object are distinguished based on facial recognition, and the distance between the center of the mobile display and the center of the user's eyebrows is used to determine the moving radius; the moving radius is determined according to the average value of the distance between the center of the mobile display and the center of the user's eyebrows in the latest use time T after the user turns on the device this time, and the distance is taken every period T1, T1 is less than T, if the use time after turning on the device is less than T, the use time is directly selected, if it is less than T1, it is not selected;

[0012] After determining the moving radius, the intersection of the vertical line of the eyebrow center and the ground is taken as the first moving center, and the moving area on the ground is formed according to the moving radius, and the data integration platform of the mobile display is fed back to obtain the object coordinate data existing in the moving area on the ground, which is marked as an obstacle avoidance point.

[0013] According to the above technical scheme, it also includes:

[0014] Obtain the video data information played by the mobile display, analyze the action data of the characters in the video, and form a video character action data set;

[0015] Use the plurality of sensors arranged around the mobile display to identify the action data of the user, form a video frame under a set fixed period, and compare the action data of the video frame with the action data of the video character action data set to obtain the action data similarity of the user at different times.

[0016] According to the above technical scheme, in steps S2 to S3, it also includes:

[0017] Construct a historical database, which stores the action data similarity of the user at different times, and the rotation angle and user behavior corresponding to the similarity;

[0018] The user behavior includes positive feedback behavior and non-positive feedback behavior, the positive feedback behavior refers to the user watching the display after a certain similarity; in this application, the positive feedback behavior actually refers to the user turning around or looking at the display when the rotation angle reaches a certain degree, that is, when the user moves the display with his back or sideways, which means that the user is not familiar with the video action and needs to continue watching the display to learn, which is the positive feedback behavior; the non-positive feedback behavior refers to the user not watching the display;

[0019] The rotation angle refers to the angle between the positive direction of the user's eyebrow center and the initial direction, the positive direction of the user's eyebrow center refers to the ray formed by taking the eyebrow center as the vertex and the front as the positive direction; the initial direction refers to the positive direction of the eyebrow center when the user initially watches the display, and the angle is 0 when the user initially watches;

[0020] The data in the historical database is selected and formed into a data combination, denoted as [Si , θ i , A0], wherein S0 represents the action data similarity of the user at time i, θ i represents the rotation angle at time i, and A0 represents the user behavior in the time range [i, i+u]; u represents a constant term set by the system.

[0021] An integrated analysis processing function is constructed to analyze the probabilities of the positive feedback behavior group and the non-positive feedback behavior group under different user action data similarities and rotation angles.

[0022] The data is divided into two groups, namely the positive feedback behavior group and the non-positive feedback behavior group, and any kind of behavior analysis is as follows:

[0023] The different user action data similarities and rotation angles are written in coordinate form, the different user action data similarities are taken as the horizontal coordinates, the rotation angles are taken as the vertical coordinates, linear fitting is performed, linear fitting functions of the positive feedback behavior group and the non-positive feedback behavior group are formed, the current user action data similarity and rotation angle are taken, the distances from the linear fitting functions of the positive feedback behavior group and the non-positive feedback behavior group are calculated respectively, and the smaller distance is selected as the output result. The behavior group corresponding to the output result is recorded as the current behavior.

[0024] When the current behavior belongs to the positive feedback behavior, the instruction analysis is started.

[0025] The instruction includes a movement start instruction and a movement termination instruction. The movement start instruction indicates that the movement is started in the positive direction of the center of the eyebrow of the current user in the initial direction, and the obstacle avoidance processing is realized during the movement. The movement termination instruction indicates that the movement is stopped when the first rotation angle is within the range set by the system. The first rotation angle refers to the difference between the rotation angle and the angle of movement under the movement start instruction. In this application, due to the movement of the display, the difference of the rotation angle gradually decreases, and when it is within a certain difference range, it is within the user's field of view, and the termination instruction is issued.

[0026] A display data integrated processing system based on multi-sensor fusion, the system includes a sensor data acquisition module, a mobile display data integration module, an instruction processing module, and a display module.

[0027] The sensor data collection module is used for identifying user behavior actions and mobile display peripheral environment data when a plurality of sensors are arranged around the mobile display, and feeding back to the mobile display data integration module; the mobile display data integration module is used for building a mobile display data integration platform, integrating and analyzing data by using the mobile display data integration platform, and forming a mobile start instruction and a mobile termination command; the instruction processing module controls the mobile display to intelligently move based on the mobile start instruction and the mobile display peripheral environment data, and stops moving when the mobile termination command is reached; the display module is used for identifying user behavior actions again, and controlling the mobile display to continue video display.

[0028] The output end of the sensor data collection module is connected with the input end of the mobile display data integration module; the output end of the mobile display data integration module is connected with the input end of the instruction processing module; and the output end of the instruction processing module is connected with the input end of the display module.

[0029] According to the above technical solution, the sensor data collection module includes a user behavior identification unit and an environment identification unit.

[0030] The user behavior identification unit is used for identifying user behavior actions; and the environment identification unit is used for identifying mobile display peripheral environment data.

[0031] The user behavior identification unit and the environment identification unit are both connected to the input end of the mobile display data integration module.

[0032] According to the above technical solution, the mobile display data integration module includes an integrated analysis processing unit and an instruction output unit.

[0033] The integrated analysis processing unit is used for building a mobile display data integration platform, and integrating and analyzing data by using the mobile display data integration platform; and the instruction output unit is used for judging whether a mobile start instruction and a mobile termination command are formed, and outputting to the instruction processing module after being formed.

[0034] The output end of the integrated analysis processing unit is connected with the input end of the instruction output unit.

[0035] According to the above technical solution, the instruction processing module includes instruction control.

[0036] The mobile start instruction means starting to move in the direction of the current user's eyebrow center, and realizing obstacle avoidance processing in the moving process; and the mobile termination command means stopping when a first rotation angle is within a system set range, and the first rotation angle means the difference between a rotation angle and the angle of moving under the mobile start instruction.

[0037] The rotation angle refers to the angle between the user's brow center direction and the initial direction. The user's brow center direction refers to the ray formed with the brow center as the vertex and the front of the user as the positive direction. The initial direction refers to the positive direction of the brow center when the user initially views the monitor, and the angle is 0 when the user initially views the monitor.

[0038] The instruction processing module controls the mobile display to move intelligently based on the start-of-movement command and the surrounding environment data of the mobile display, and stops moving when the end-of-movement command is reached.

[0039] According to the above technical solution, the display module includes a user action comparison unit and a display unit;

[0040] After the movement termination command ends, the user action comparison unit re-identifies the current user action; the display unit is used to feed back the re-identified user action to the mobile display to continue video display.

[0041] The output of the user action comparison unit is connected to the input of the display unit.

[0042] Compared with the prior art, the beneficial effects of the present invention are: the present invention mainly solves the visual problems of mobile displays in fitness and learning scenarios. Based on the integrated processing of sensor data, it forms intelligent instruction analysis, which solves the problem that users have difficulty viewing the content at the same time when they need to turn to the side or turn their back to the mobile display due to different movements of fitness content in fitness and learning scenarios, thus promoting the further intelligent level of mobile displays. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the display data integration and processing method based on multi-sensor fusion according to the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Example: Figure 1 As shown in this embodiment, the present invention provides a display data integration and processing method based on multi-sensor fusion. The method includes: deploying a number of sensors around the mobile display to identify user behavior and data of the surrounding environment of the mobile display, and feeding them back to the mobile display data integration platform.

[0046] The data used to identify user behavior and the surrounding environment of the mobile display includes:

[0047] Based on the mobile display peripheral layout of several sensors, with the mobile display moving rod center point as the center, the horizontal and vertical directions form the space rectangular coordinate system in the monitoring area, monitor the objects in the area, mark the object coordinates;

[0048] Based on facial recognition to distinguish users and objects, and the distance between the mobile display center and the user's eyebrow center is used to determine the moving radius; the moving radius is determined according to the average value of the distance between the mobile display center and the user's eyebrow center within the last use time T after the user's current boot, and the distance is taken every period T1, T1 is less than T, if the use time after boot is less than T, then directly select according to the use time, if less than T1, do not select;

[0049] After determining the moving radius, the intersection of the eyebrow center perpendicular to the ground and the ground is taken as the first moving center, and the moving area on the ground is formed according to the moving radius, and the data integration platform of the mobile display is fed back to obtain the object coordinate data existing in the moving area on the ground, which is marked as an obstacle avoidance point.

[0050] The mobile display data integration platform integrates, analyzes and processes the data to form a moving start instruction and a moving termination instruction; based on the moving start instruction and the peripheral environment data of the mobile display, the mobile display is controlled to move intelligently, and stops moving when the moving termination instruction is reached;

[0051] Obtain the video data information played by the mobile display, analyze the character actions in the video, and form a video character action data set;

[0052] Use the several sensors arranged peripherally on the mobile display to identify the user's action data, form the video frame under the set fixed period, compare the video frame action data with the action data of the video character action data set, and obtain the action data similarity of the user at different times.

[0053] Construct a large historical database, which stores the action data similarity of the user at different times, and the rotation angle and user behavior corresponding to the similarity;

[0054] The user behavior includes positive feedback behavior and non-positive feedback behavior, the positive feedback behavior refers to the user watching the display after a certain similarity; the non-positive feedback behavior refers to the user not watching the display;

[0055] The rotation angle refers to the included angle between the positive direction of the user's eyebrow center and the initial direction; the positive direction of the user's eyebrow center refers to the ray formed with the eyebrow center as the vertex and the front as the positive direction; the initial direction refers to the positive direction of the eyebrow center when the user initially watches the display, and the included angle is 0 when the user initially watches;

[0056] For data selection and combination in a historical large database, denoted as [S i , θ i , A0], wherein S0 represents the action data similarity of the user at time i, θ i represents the rotation angle at time i, and A0 represents the user behavior in the time range [i, i+u]; u represents a constant term set by the system;

[0057] An integrated analysis processing function is constructed to analyze the probabilities of the positive feedback behavior group and the non-positive feedback behavior group under different user action data similarity and rotation angles;

[0058] The data is divided into two groups, namely the positive feedback behavior group and the non-positive feedback behavior group, and any kind of behavior analysis is as follows:

[0059] The different user action data similarity and rotation angles are written in coordinate form, the user action data similarity is taken as the horizontal coordinate, the rotation angle is taken as the vertical coordinate, linear fitting is performed, the linear fitting functions of the positive feedback behavior group and the non-positive feedback behavior group are formed, the action data similarity and the rotation angle of the current user are calculated respectively, the distance from the linear fitting functions of the positive feedback behavior group and the non-positive feedback behavior group is calculated, and the smaller distance is selected as the output result; the behavior group corresponding to the output result is denoted as the current behavior;

[0060] When the current behavior belongs to the positive feedback behavior, the instruction analysis is started;

[0061] The instruction includes a movement start instruction and a movement termination command, the movement start instruction indicates that the movement is started in the positive direction of the center of the eyebrow of the current user along the initial direction, and the obstacle avoidance processing is realized during the movement; the movement termination command indicates that the movement is stopped when the first rotation angle is within the range set by the system, and the first rotation angle refers to the difference between the rotation angle and the angle of the movement under the movement start instruction.

[0062] After the movement termination command is ended, the user behavior action is recognized again, and the mobile display is controlled to continue the video display.

[0063] In the embodiment, a display data integrated processing system based on multi-sensor fusion is also provided, which includes a sensor data acquisition module, a mobile display data integration module, an instruction processing module, and a display module.

[0064] The sensor data acquisition module is used to deploy several sensors around the mobile display to identify user behavior and data from the surrounding environment, and feed this data back to the mobile display data integration module. The mobile display data integration module is used to build a mobile display data integration platform, which integrates and analyzes the data to generate a start and stop command. The command processing module controls the mobile display to move intelligently based on the start command and the surrounding environment data, stopping the movement when the stop command is reached. The display module is used to re-identify user behavior and control the mobile display to continue displaying video.

[0065] The output of the sensor data acquisition module is connected to the input of the mobile display data integration module; the output of the mobile display data integration module is connected to the input of the instruction processing module; and the output of the instruction processing module is connected to the input of the display module.

[0066] The sensor data acquisition module includes a user behavior recognition unit and an environment recognition unit;

[0067] The user behavior recognition unit is used to recognize user behavior actions; the environment recognition unit is used to recognize environmental data around the mobile display.

[0068] Both the user behavior recognition unit and the environment recognition unit are connected to the input terminal of the mobile display data integration module.

[0069] The mobile display data integration module includes an integrated analysis and processing unit and an instruction output unit;

[0070] The integrated analysis and processing unit is used to build a mobile display data integration platform and use the mobile display data integration platform to perform integrated analysis and processing on the data; the instruction output unit is used to determine whether a movement start instruction and a movement termination command are generated, and after they are generated, they are output to the instruction processing module.

[0071] The output of the integrated analysis and processing unit is connected to the input of the instruction output unit.

[0072] The instruction processing module includes instruction control:

[0073] The move start command refers to starting to move along the initial direction towards the current user's brow, and performing obstacle avoidance during the movement; the move stop command refers to stopping when the first rotation angle is within the range set by the system, where the first rotation angle is the difference between the rotation angle and the angle moved under the move start command.

[0074] The rotation angle refers to the included angle between the user's eyebrow center positive direction and the initial direction, the user's eyebrow center positive direction refers to the ray formed by taking the eyebrow center as the vertex and the front as the positive direction; the initial direction refers to the positive direction of the eyebrow center when the user initially watches the display, and the included angle is 0 when the user initially watches;

[0075] The instruction processing module controls the mobile display to intelligently move based on the movement start instruction and the mobile display peripheral environment data, and stops moving when the movement termination instruction is reached.

[0076] The display module comprises a user action comparison unit and a display unit.

[0077] The user action comparison unit re-recognizes the current user action after the movement termination instruction ends, and the display unit is used for feeding back to the mobile display according to the re-recognized user action to continue video display.

[0078] The output end of the user action comparison unit is connected with the input end of the display unit.

[0079] It is apparent to those skilled in the art that the present application is not limited to the details of the foregoing exemplary embodiments, and can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the foregoing description, and all changes falling within the meaning and range of the equivalent elements of the claims are intended to be encompassed by the present application. Any reference signs in the claims should not be considered as limiting the claims involved.

Claims

1. A display data integration processing method based on multi-sensor fusion, characterized in that: The method comprises the following steps: S1, a plurality of sensors are arranged around the mobile display, which are used to identify user behavior actions and mobile display peripheral environment data, and feed back to the mobile display data integration platform; S2, the mobile display data integration platform integrates, analyzes and processes the data to form a mobile start instruction and a mobile termination instruction; S3, the mobile display is controlled to intelligently move based on the mobile start instruction and the mobile display peripheral environment data, and stops moving when the mobile termination instruction is reached; S4, the user behavior actions are identified, and the mobile display is controlled to continue video display.

2. The multi-sensor fusion based display data integration processing method according to claim 1, characterized in that: The identification of user behavior actions and mobile display peripheral environment data comprises: Based on the plurality of sensors arranged around the mobile display, a space rectangular coordinate system is formed in the region with the mobile display moving rod center point as the center and in the horizontal and vertical directions, objects in the region are monitored, and object coordinates are marked; Based on face recognition, the user and the object are distinguished, and the distance between the mobile display center and the user's eyebrow center is used to determine the moving radius; the moving radius is determined according to the average value of the distance between the mobile display center and the user's eyebrow center within the latest use time T after the user turns on the machine, the distance is taken every period T1, T1 is less than T, if the use time after turning on the machine is less than T, the use time is directly selected, if it is less than T1, it is not selected; After the moving radius is determined, the intersection of the eyebrow center perpendicular to the ground and the ground is taken as the first moving center, the moving area of the ground is formed according to the moving radius, and the moving area of the ground is fed back to the mobile display data integration platform to obtain the object coordinate data existing in the moving area of the ground, which is marked as an obstacle avoidance point.

3. The multi-sensor fusion based display data integration processing method according to claim 2, characterized in that: Further comprising: Obtaining video data information played by the mobile display, analyzing the character actions in the video to form a video character action data set; Using the plurality of sensors arranged around the mobile display to identify the user's action data, forming a video frame under a set fixed period, comparing the video frame action data with the action data of the video character action data set, and obtaining the action data similarity of the user at different times.

4. The multi-sensor fusion based display data integration processing method according to claim 3, characterized in that: In steps S2 to S3, further comprising: Building a historical database, the historical database stores the action data similarity of the user at different times, and the rotation angle and user behavior corresponding to the similarity; The user behavior includes positive feedback behavior and non-positive feedback behavior, the positive feedback behavior refers to the user watching the display after a certain similarity; the non-positive feedback behavior refers to the user not watching the display; The rotation angle refers to the included angle between the positive direction of the user's eyebrow center and the initial direction, the positive direction of the eyebrow center refers to the ray formed with the eyebrow center as the vertex and the front as the positive direction; the initial direction refers to the positive direction of the eyebrow center when the user initially watches the display, the included angle is 0 when the user initially watches; For data selection and formation of data combination in a historical large database, denoted as [S i , θ i , A0], wherein S0 represents the action data similarity of the user at time i, θ i represents the rotation angle at time i, and A0 represents the user behavior in the time range [i, i+u]; u represents a constant term set by the system; Building an integrated analysis and processing function to analyze the probability of the occurrence of positive feedback behavior and non-positive feedback behavior groups under different user action data similarity and rotation angle; The data is divided into two groups, namely positive feedback behavior group and non-positive feedback behavior group, and any one behavior is analyzed as follows: The action data similarity and the rotation angle of different users are written in coordinate form, the action data similarity of different users is taken as the horizontal coordinate, the rotation angle is taken as the vertical coordinate, linear fitting is performed, linear fitting functions of the positive feedback behavior group and the non-positive feedback behavior group are formed, the action data similarity and the rotation angle of the current user are taken, the distance from the linear fitting functions of the positive feedback behavior group and the non-positive feedback behavior group is calculated respectively, and the smaller distance is taken as the output result, and the behavior group corresponding to the output result is recorded as the current behavior; When the current behavior belongs to the positive feedback behavior, the instruction analysis is started; The instruction includes a movement start instruction and a movement termination instruction, the movement start instruction indicates that the movement in the positive direction of the eyebrow center of the current user is started in the initial direction, and the obstacle avoidance processing is realized in the movement process; the movement termination instruction indicates that the movement is stopped when the first rotation angle is in the range set by the system, and the first rotation angle indicates the difference between the rotation angle and the angle of the movement under the movement start instruction.

5. A display data integration processing system based on multi-sensor fusion, characterized by: The system comprises a sensor data acquisition module, a mobile display data integration module, an instruction processing module, and a display module. The sensor data acquisition module is used to identify user behavior actions and mobile display peripheral environment data, and feed back to the mobile display data integration module; the mobile display data integration module is used to build a mobile display data integration platform, integrate and analyze the data by using the mobile display data integration platform, and form a movement start instruction and a movement termination instruction; the instruction processing module controls the intelligent movement of the mobile display based on the movement start instruction and the mobile display peripheral environment data, and stops moving when the movement termination instruction is reached; the display module is used to identify user behavior actions again, and control the mobile display to continue video display; The output end of the sensor data acquisition module is connected with the input end of the mobile display data integration module; the output end of the mobile display data integration module is connected with the input end of the instruction processing module; and the output end of the instruction processing module is connected with the input end of the display module.

6. The multi-sensor fusion based display data integration processing system of claim 5, wherein: The sensor data acquisition module comprises a user behavior recognition unit and an environment recognition unit; The user behavior recognition unit is used to identify user behavior actions; and the environment recognition unit is used to identify mobile display peripheral environment data; The user behavior recognition unit and the environment recognition unit are both connected to the input end of the mobile display data integration module.

7. The multi-sensor fusion based display data integration processing system of claim 5, wherein: The mobile display data integration module comprises an integrated analysis processing unit and an instruction output unit; The integrated analysis processing unit is used to build a mobile display data integration platform, and integrate and analyze the data by using the mobile display data integration platform; and the instruction output unit is used to determine whether to form a movement start instruction and a movement termination instruction, and output to the instruction processing module after formation; The output end of the integrated analysis processing unit is connected with the input end of the instruction output unit.

8. The multi-sensor fusion based display data integration processing system of claim 5, wherein: The instruction processing module comprises instruction control: The movement start instruction indicates to start moving in the initial direction to the center of the user's eyebrows, and to realize obstacle avoidance processing during the movement; the movement end instruction indicates to stop when the first rotation angle is within the range set by the system, and the first rotation angle indicates the difference between the rotation angle and the angle of movement after the movement start instruction; The rotation angle indicates the included angle between the center of the user's eyebrows and the initial direction, the center of the user's eyebrows indicates the ray formed with the center of the eyebrows as the vertex and the front as the positive direction; the initial direction indicates the positive direction of the center of the eyebrows when the user initially watches the display, and the included angle is 0 when the user initially watches; The instruction processing module controls the intelligent movement of the mobile display based on the movement start instruction and the movement display peripheral environment data, and stops moving when the movement end instruction is reached.

9. The multi-sensor fusion based display data integration processing system of claim 5, wherein: The display module includes a user action comparison unit and a display unit; The user action comparison unit re-identifies the current user action after the movement end instruction ends; the display unit is used to feed back to the mobile display according to the re-identified user action, and continue to display the video; The output end of the user action comparison unit is connected to the input end of the display unit.

Images