Object monitoring system and method and electronic equipment

Abstract

The embodiment of the invention provides an object monitoring system and method and electronic equipment, and belongs to the technical field of data monitoring and the technical field of data analysis. The method comprises the following steps: acquiring target multi-modal data of a target object through a multi-modal data acquisition module, and transmitting the target multi-modal data to a data processing module; then, performing motion behavior analysis on the target object through a data processing module to obtain motion behavior analysis data; and carrying out physiological state change analysis on the target object to obtain physiological state change analysis data. Finally, the data processing module transmits the exercise behavior analysis data and the physiological state change analysis data to a display module, and the exercise behavior analysis data and the physiological state change analysis data are displayed through the display module. Therefore, the object monitoring system provided by the invention can collect and analyze the target multi-modal data of the target object, the monitoring comprehensiveness of the object monitoring system is improved, and the time cost of data collection and analysis is reduced.

Description

Technical Field

[0001] This application relates to the technical fields of data monitoring and data analysis, and particularly to an object monitoring system, method, and electronic device. Background Art

[0002] Object monitoring systems play a very important role in the work of collecting object phenotypic data. Currently, object monitoring systems are mainly used to monitor and collect the physiological data of objects, including the feeding, drinking habits, activity levels, respiratory rates, etc. of objects, and are widely used in the exploration of other life science fields such as kinematics, physiology, and pathology.

[0003] However, existing object monitoring systems focus on physiological data collection and lack attention to other phenotypic data. When researchers need to collect and analyze multi-faceted phenotypic data of an object, and an object monitoring system can only achieve the collection and analysis of data in a single aspect, it is often necessary to replace different object monitoring systems to collect and analyze the multi-faceted phenotypic data of the object, which will increase the time cost of data collection and analysis. Therefore, how to improve the comprehensiveness of data analysis of object monitoring systems and reduce the time cost of data collection and analysis has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose an object monitoring system, method, and electronic device, aiming to improve the comprehensiveness of data collection and analysis of object monitoring systems and reduce the time cost of data collection and analysis.

[0005] To achieve the above object, the first aspect of the embodiments of this application proposes an object monitoring system, and the system includes:

[0006] A multi-modal data collection module for collecting target multi-modal data of a target object; wherein, the target multi-modal data includes: monitoring video data and sensor data;

[0007] A data processing module communicatively connected to the multi-modal data collection module, and the data processing module is used for:

[0008] Performing motion behavior analysis on the target object based on the monitoring video data to obtain motion behavior analysis data; wherein, the monitoring video data represents the video data of the motion process of the target object;

[0009] Performing physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data; wherein, the sensor data represents the physiological data of the life process of the target object;

[0010] A display module, electrically connected to the data processing module, for displaying the motion behavior analysis data and the physiological state change analysis data.

[0011] In some embodiments, the data processing module includes: a video analysis unit, a motion behavior analysis unit, and a physiological state change analysis unit;

[0012] The video analysis unit is configured to:

[0013] Split the monitored video data to obtain a plurality of monitored video frame images;

[0014] Input the plurality of monitored video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data; wherein, the motion pose estimation data characterizes the motion data of each body point of the target object;

[0015] The motion behavior analysis unit is configured to:

[0016] Perform motion behavior analysis based on the motion pose estimation data to obtain the motion behavior analysis data;

[0017] The physiological state change analysis unit is configured to:

[0018] Extract physiological data of the target object based on the sensor data to obtain physiological data;

[0019] Perform physiological state change analysis on the target object based on the physiological data to obtain the physiological state change analysis data.

[0020] In some embodiments, the motion pose estimation data includes: body point motion data and action data, the motion behavior analysis data includes: motion analysis data and behavior analysis data, the motion analysis data includes: motion parameter distribution data, motion trajectory data, motion position data, and kinematic parameter inter-group data, the behavior analysis data includes: action clustering analysis data, inter-group action distribution data, action distribution statistical data, and action spectrum; the motion behavior analysis unit includes: a motion analysis subunit and a behavior analysis subunit;

[0021] The motion analysis subunit is configured to:

[0022] Perform parameter distribution analysis on the target object based on the body point motion data to obtain the motion parameter distribution data;

[0023] Perform motion trajectory analysis on the target object based on the body point motion data to obtain the motion trajectory data;

[0024] Performing motion position analysis on the target object based on the body point motion data to obtain the motion position data;

[0025] Performing inter-group kinematic parameter analysis on the target object based on the body point motion data and a preset grouping of target objects to obtain the inter-group kinematic parameter data;

[0026] The behavior analysis subunit is used for:

[0027] Performing clustering analysis on the action data to obtain the action clustering analysis data;

[0028] Performing inter-group action distribution analysis on the target object based on the action data to obtain the inter-group action distribution data;

[0029] Performing action distribution statistical analysis on the target object based on the action data to obtain the action distribution statistical data;

[0030] Constructing an action spectrum according to the action data.

[0031] In some embodiments, the multi-modal data acquisition module includes: a data acquisition control device, a video data acquisition device, and a sensor data acquisition device;

[0032] The data acquisition control device is used for:

[0033] Obtaining preset target object information and target data acquisition information; wherein, the target object information is used to represent the basic information of the target object, and the target data acquisition information is used to represent the information of the data to be acquired;

[0034] Controlling the video data acquisition device and the sensor data acquisition device to perform data acquisition on the target object according to a preset data acquisition rule, the target object information, and the target data acquisition information to obtain candidate multi-modal data;

[0035] Screening the candidate multi-modal data according to a preset information transmission rule to obtain the target multi-modal data.

[0036] To achieve the above object, a second aspect of the embodiments of the present application proposes an object monitoring method, and the method includes:

[0037] Obtaining target multi-modal data; wherein, the target multi-modal data is data collected by a multi-modal data acquisition module during the movement process and life process of a target object, and the target multi-modal data includes: monitoring video data and sensor data;

[0038] Performing motion behavior analysis on the target object based on the monitored video data to obtain motion behavior analysis data; wherein, the monitored video data represents video data of the motion process of the target object;

[0039] Performing physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data; wherein, the sensor data represents physiological data of the life process of the target object;

[0040] Sending the motion behavior analysis data and the physiological state change analysis data to a display module so that the display module displays the motion behavior analysis data and the physiological state change analysis data.

[0041] In some embodiments, the performing motion behavior analysis on the target object based on the monitored video data to obtain motion behavior analysis data includes:

[0042] Performing splitting processing on the monitored video data to obtain a plurality of monitored video frame images;

[0043] Inputting the plurality of monitored video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data; wherein, the motion pose estimation data represents motion data on each body point of the target object;

[0044] Performing motion behavior analysis based on the motion pose estimation data to obtain the motion behavior analysis data.

[0045] In some embodiments, the motion pose estimation data includes: body point motion data and action data, the motion behavior analysis data includes: motion analysis data and behavior analysis data, the motion analysis data includes: motion parameter distribution data, motion trajectory data, motion position data, and kinematic parameter inter-group data, the behavior analysis data includes: action clustering analysis data, inter-group action distribution data, action distribution statistical data, and action spectrum; the performing motion behavior analysis based on the motion pose estimation data to obtain the motion behavior analysis data includes:

[0046] Performing parameter distribution analysis on the target object based on the body point motion data to obtain the motion parameter distribution data;

[0047] Performing motion trajectory analysis on the target object based on the body point motion data to obtain the motion trajectory data;

[0048] Performing motion position analysis on the target object based on the body point motion data to obtain the motion position data;

[0049] Performing an inter-group analysis of kinematic parameters on the target object based on the body point motion data and a preset grouping of target objects to obtain the inter-group data of kinematic parameters;

[0050] Performing a clustering analysis on the action data to obtain the action clustering analysis data;

[0051] Performing an inter-group action distribution analysis on the target object based on the action data to obtain the inter-group action distribution data;

[0052] Performing an action distribution statistical analysis on the target object based on the action data to obtain the action distribution statistical data;

[0053] Constructing an action spectrum according to the action data.

[0054] In some embodiments, the performing a physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data includes:

[0055] Performing a physiological data extraction on the target object based on the sensor data to obtain physiological data;

[0056] Performing a physiological state change analysis on the target object based on the physiological data to obtain the physiological state change analysis data.

[0057] In some embodiments, before obtaining the target multi-modal data of the target object, the method further includes:

[0058] Obtaining preset target object information and target data collection information; wherein, the target object information is used to represent the basic information of the target object, and the target data collection information is used to represent the information of the data to be collected;

[0059] Performing data collection on the target object according to a preset data collection rule, the target object information, and the target data collection information to obtain candidate multi-modal data;

[0060] Filtering the candidate multi-modal data according to a preset information transmission rule to obtain the target multi-modal data.

[0061] To achieve the above object, a third aspect of the embodiments of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method described in the second aspect above is implemented.

[0062] The object monitoring system, method, and electronic device proposed in this application first collect target multi-modal data of a target object through a multi-modal data acquisition module; among them, the target multi-modal data includes: surveillance video data and sensor data. Then, the multi-modal data acquisition module and the data processing module are connected through a communication connection, and the target multi-modal data is transmitted to the data processing module. The data processing module analyzes the motion behavior of the target object based on the surveillance video data to obtain motion behavior analysis data, where the surveillance video data represents the video data of the motion process of the target object; and analyzes the physiological state changes of the target object based on the sensor data to obtain physiological state change analysis data, where the sensor data represents the physiological data of the life process of the target object. Finally, the data processing module and the display module are electrically connected, and the motion behavior analysis data and the physiological state change analysis data are transmitted to the display module, and the motion behavior analysis data and the physiological state change analysis data are displayed through the display module. Therefore, the object monitoring system, method, and electronic device proposed in this application can collect the target multi-modal data of the target object, analyze the collected target multi-modal data, can obtain both the motion behavior analysis data and the physiological state change analysis data at the same time, improve the comprehensiveness of data collection and analysis of the object monitoring system, and reduce the time required for collecting and analyzing data by replacing the object monitoring system multiple times, thereby reducing the time cost of data collection and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is an optional structural schematic diagram of the object monitoring system provided by an embodiment of the present application;

[0064] Figure 2 is an optional structural schematic diagram of the multi-modal data acquisition module provided by an embodiment of the present application;

[0065] Figure 3 is another optional structural schematic diagram of the multi-modal data acquisition module provided by an embodiment of the present application;

[0066] Figure 4 is an optional structural schematic diagram of the data processing module provided by an embodiment of the present application;

[0067] Figure 5 is an optional structural schematic diagram of the motion behavior analysis unit provided by an embodiment of the present application;

[0068] Figure 6 is another optional structural schematic diagram of the object monitoring system provided by an embodiment of the present application;

[0069] Figure 7 is an optional flowchart of the object monitoring method provided by an embodiment of the present application;

[0070] Figure 8 is Figure 7 the flowchart of step S702 in

[0071] Figure 9 is Figure 8 the flowchart of step S803 in

[0072] Figure 10 is Figure 7 the flowchart of step S703 in

[0073] Figure 11 is another optional flowchart of the object monitoring method provided by the embodiments of the present application;

[0074] Figure 12 is an optional overall flowchart of the object monitoring method provided by the embodiments of the present application;

[0075] Figure 13 is a schematic diagram of the hardware structure of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0076] In order to make the purpose, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0077] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the order in the flowchart.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0079] In the fields of life and health such as biomedicine and neurobehavior, it is crucial to collect accurate and comprehensive phenotypic data of the target object. The phenotypic data of the target object covers multiple aspects such as the object's behavior patterns, physiological conditions, and adaptability to environmental changes. They are of inestimable value for deeply understanding the health status, behavioral characteristics, and response patterns to specific stimuli of the target object. Traditional target object monitoring systems are mainly used for monitoring physiological data. These physiological data can be applied to the research of endocrine and metabolic diseases such as nutrition, obesity, and diabetes. However, when studying experiments involving cognition, memory, etc., it is also necessary to collect and analyze other phenotypic data of the target object such as motion data and behavioral data. To complete the above research involving experiments such as cognition and memory, it is usually necessary to replace different experimental devices to collect and analyze the phenotypic data of different aspects of the target object, which will increase the time cost of data collection and analysis. Therefore, how to improve the comprehensiveness of data collection and analysis of the target object monitoring system and reduce the time cost of data collection and analysis has become an urgent technical problem to be solved.

[0080] Based on this, the embodiments of the present application provide an object monitoring system, method, and electronic device, aiming to improve the comprehensiveness of data collection and analysis of the target object monitoring system and reduce the time cost of data collection and analysis.

[0081] The object monitoring system, method, and electronic device provided by the embodiments of the present application are specifically described through the following embodiments. First, the object monitoring system in the embodiments of the present application is described.

[0082] Figure 1 is an optional structural schematic diagram of the object monitoring system provided by the embodiments of the present application. Figure 1 The system in may include but is not limited to: a multi-modal data acquisition module 100, a data processing module 200, and a display module 300.

[0083] The multi-modal data acquisition module 100 is used to acquire the target multi-modal data of the target object; among them, the target multi-modal data includes: surveillance video data and sensor data.

[0084] The data processing module 200 is communicatively connected to the multi-modal data acquisition module 100, and the data processing module 200 is used for:

[0085] Performing motion behavior analysis on the target object based on the surveillance video data to obtain motion behavior analysis data; among them, the surveillance video data represents the video data of the target object's motion process.

[0086] Performing physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data; among them, the sensor data represents the physiological data of the target object's life process.

[0087] The display module 300 is electrically connected to the data processing module 200 and is used to display the motion behavior analysis data and the physiological state change analysis data.

[0088] In the object monitoring system shown in the embodiments of the present application, first, the multi-modal data acquisition module 100 acquires the target multi-modal data of the target object; among them, the target multi-modal data includes: monitoring video data and sensor data. Then, the multi-modal data acquisition module 100 and the data processing module 200 are connected through a communication connection, and the target multi-modal data is transmitted to the data processing module 200. The data processing module 200 performs motion behavior analysis on the target object based on the monitoring video data to obtain motion behavior analysis data, and performs physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data. Finally, the data processing module 200 and the display module 300 are electrically connected, and the motion behavior analysis data and the physiological state change analysis data are transmitted to the display module 300, and the motion behavior analysis data and the physiological state change analysis data are displayed through the display module 300. Therefore, the object monitoring system proposed in the present application can acquire the target multi-modal data of the target object, analyze the acquired target multi-modal data, can simultaneously obtain the motion behavior analysis data and the physiological state change analysis data, realize a more comprehensive analysis of the target object, and use the multi-modal data acquisition module 100 to realize the acquisition of the object multi-modal data, without having to replace the object monitoring system multiple times for data acquisition and analysis, which can reduce the time required for data acquisition and analysis and improve the data analysis efficiency of the target object in all aspects.

[0089] In some embodiments, the size of the multi-modal data acquisition module 100 can be: 45 cm in length, 40 cm in width, and 20 cm in height, or it can be customized according to the needs of those skilled in the art.

[0090] Please refer to Figure 2 , in some embodiments, the multi-modal data acquisition module 100 includes but is not limited to: a data acquisition control device 110, a video data acquisition device 120, and a sensor data acquisition device 130;

[0091] The data acquisition control device 110 is used for:

[0092] acquiring the preset target object information and target data acquisition information; among them, the target object information is used to represent the basic information of the target object, and the target data acquisition information is used to represent the information of the data that needs to be acquired;

[0093] controlling the video data acquisition device 120 and the sensor data acquisition device 130 to acquire data from the target object according to the preset data acquisition rules, target object information, and target data acquisition information, and obtaining candidate multi-modal data;

[0094] Filter the candidate multimodal data according to the preset information transmission rules to obtain the target multimodal data.

[0095] Figure 3 It is another optional structural schematic diagram of the multimodal data acquisition module provided by the embodiments of the present application, which describes the overall structure after the assembly of each module. As Figure 2 and Figure 3 shown, the multimodal data acquisition module 100 shown in the embodiments of the present application sets a modular cage and a video data acquisition device 120 within a structured framework, and sets within the modular cage: a data acquisition control device 110, a sensor data acquisition device 130, bedding, and activity facilities. Multiple types of experimental data of a target object can be collected through one multimodal data acquisition module, including but not limited to: multiple types of experimental data for cognitive experiments and maze experiments. Since the activity facilities and the sensor data acquisition device 130 in the multimodal data acquisition module 100 can be added or removed according to the experiments conducted by those skilled in the art and the data to be collected, it can adapt to different experimental requirements, collect different experimental data, has good applicability, reduces the time required for collecting data by replacing the object monitoring system multiple times, and thus reduces the time cost of data collection.

[0096] In some embodiments, the structured framework is a detachable framework, and the material for making the structured framework can be aluminum or other materials that can be used to make the structured framework for object experiments.

[0097] In some embodiments, the data acquisition control device 110 may include but is not limited to: a main control unit. Among them, the main control unit can be a Raspberry Pi microcomputer, or other single-chip microcomputers, or a combination of multiple single-chip microcomputers, as long as it has the following functions: obtaining the preset target object information and target data acquisition information, and controlling the video data acquisition device 120 and the sensor data acquisition device 130 to collect data from the target object according to the preset data acquisition rules, target object information, and target data acquisition information, to obtain candidate multimodal data, and then filtering the candidate multimodal data according to the preset information transmission rules to obtain the target multimodal data. Since the object monitoring system shown in the embodiments of the present application and the object monitoring method shown in the embodiments of the present application are basically the same, the specific data acquisition rules can refer to the content related to the data acquisition rules in the object monitoring method shown in the embodiments of the present application, and will not be elaborated here. The target object information is used to characterize the characteristics of the target object. If the target object is an animal, then the characteristics of the target object include at least one of the following: animal species, body size, gender, age group, group, etc. The target data acquisition information is used to characterize the monitoring parameters and their thresholds, and those skilled in the art can set the target data acquisition information according to experimental requirements.

[0098] In some embodiments, the video data acquisition device 120 may include, but is not limited to, any of the following devices: a fill light and an infrared camera. The fill light and the infrared camera are respectively connected to the data acquisition control device 110, and can be replaced with other video data acquisition devices that those skilled in the art can obtain. Among them, the number of infrared cameras is at least two. The fill light is used to supplement light for the infrared camera according to a preset fill light rule. The fill light can be selected according to the type of the video data acquisition device. For example, an infrared lamp can be selected to supplement light for the infrared camera. The fill light rule can be to turn on the infrared lamp to supplement light for the infrared camera during a specific period, or it can be set to turn on the infrared lamp to supplement light for the infrared camera when it is detected that the ambient brightness is lower than a specific threshold. The specific specific period or specific threshold can be set in advance, or can be customized according to the actual needs of those skilled in the art.

[0099] In some embodiments, the sensor data acquisition device 130 and the data acquisition control device 110 can be electrically connected or communicatively connected. Specifically, the sensor data acquisition device 130 may include, but is not limited to, any of the following devices: an RFID module, a weight sensor, an environmental monitoring sensor, a water spout, and a feeder. The sensor data acquisition device 130 can be added or removed according to the experiments conducted by those skilled in the art and the data to be collected. Among them, the RFID module is injected into the target object's body before the experiment starts, and is used to obtain the body temperature and position data of the target object, and transmit data to the data acquisition control device 110 through a radio frequency signal. The weight sensor is used to collect the weight data of the target object. The accuracy of the weight sensor is 0.01 g, and the weight sensor transmits data to the data acquisition control device 110 through a serial port. The environmental monitoring sensor is used to monitor the humidity data, temperature data, and air pressure data in the cage. The environmental monitoring sensor transmits data to the data acquisition control device 110 through a serial port. The water spout and the feeder are used to provide water and food for the target object, and at the same time collect the consumption data of water and food. The accuracy of the water spout and the feeder is 0.01 g, and the water spout and the feeder transmit data to the data acquisition control device 110 through a serial port.

[0100] In some embodiments, the modular cage further includes: bedding and activity facilities. Specifically, the activity facilities may include, but are not limited to, at least one of the following: a shelter, a climbing platform, a cognitive wall, a maze, and a running wheel. The specific activity facilities provided in the modular cage can be added or removed according to the experiments conducted by those skilled in the art and the data to be collected. Among them, the shelter is used for the target object to hide and build a nest; the climbing platform is used for the target object to move; the cognitive wall is used to conduct cognitive experiments on the target object; the maze is used to conduct maze experiments on the target object; the running wheel is used for the target object to move. Therefore, by providing a series of facilities for the target object to move and act in the modular cage, the collected multi-modal data of the target can more accurately characterize the movement, living, and physiological states of the target object, achieving more comprehensive monitoring.

[0101] In some embodiments, the data processing module 200 can be communicatively connected to the multi-modal data acquisition module 100. The data processing module 200 is capable of performing motion behavior analysis on the target object based on the monitoring video data to obtain motion behavior analysis data; and performing physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data.

[0102] Please refer to Figure 4 , in some embodiments, the data processing module 200 includes, but is not limited to, a video analysis unit 210, a motion behavior analysis unit 220, and a physiological state change analysis unit 230;

[0103] The video analysis unit 210 is used for:

[0104] Performing splitting processing on the monitoring video data to obtain a plurality of monitoring video frame images;

[0105] Inputting the plurality of monitoring video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data; wherein, the motion pose estimation data represents the motion data at each body point of the target object;

[0106] The motion behavior analysis unit 220 is used for:

[0107] Performing motion behavior analysis based on the motion pose estimation data to obtain motion behavior analysis data;

[0108] The physiological state change analysis unit 230 is used for:

[0109] Extracting physiological data of the target object based on the sensor data to obtain physiological data;

[0110] Performing physiological state change analysis on the target object based on the physiological data to obtain physiological state change analysis data.

[0111] In some embodiments, the video analysis unit 210 is capable of acquiring the surveillance video data transmitted by the multimodal data acquisition module 100, splitting the surveillance video data, obtaining a plurality of surveillance video frame images, and then inputting the plurality of surveillance video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data. The motion pose estimation data represents the motion data of each body point of the target object.

[0112] In some embodiments, the video analysis unit 210 further includes a pose estimation model training module. The pose estimation model training module is used to train the pose estimation model. The specific pose estimation model training process is as follows: First, acquire the surveillance video data transmitted by the multimodal data acquisition module 100, and split the surveillance video data into surveillance video candidate images according to the number of frames of the surveillance video data. Then, randomly extract the same number of pictures from the surveillance video candidate images to form surveillance video frame images. Next, set the body points of the target object to be labeled. Taking a mouse as an example, label the following 16 body points: nose, left ear, right ear, neck, left front limb, right front limb, left hind limb, right hind limb, left front paw, right front paw, left hind paw, right hind paw, center of the back, root of the tail, middle of the tail, and tip of the tail. Among them, the order of image annotation must be in the order of the configured body points. Then, collect the pixel coordinates corresponding to the body points on the pictures according to the body point arrangement order to obtain body point annotation data. Finally, input the body point annotation data into a pre-trained model for model training, and train the pose estimation model according to the preset hyperparameters to obtain the pose estimation model. The pose estimation model training module includes a plurality of pre-trained models. The pre-trained models at least include a Resnet backbone network. Those skilled in the art can select the pose estimation model training module according to actual needs. Taking the Resnet backbone network as an example, those skilled in the art can select Resnet backbone networks with different numbers of layers. The number of layers of the Resnet backbone network can be any one of the following: 50 layers, 101 layers, 152 layers. The higher the number of layers of the Resnet backbone network, the stronger its feature extraction ability, but the longer the training time. The hyperparameters include but are not limited to at least one of the following: learning rate or optimizer.

[0113] In some embodiments, the motion behavior analysis unit 220 can perform motion behavior analysis based on the motion pose estimation data to obtain motion behavior analysis data. Among them, the motion pose estimation data includes: body point motion data and action data, and the motion behavior analysis data includes: motion analysis data and behavior analysis data. Taking a mouse as an example, 39 pieces of body point motion data and 40 pieces of action data can be set. The 39 pieces of body point motion data include: body point speed data corresponding to 16 body points, body point motion intensity data corresponding to 16 body points, 1 distance data, 3 body type data, 1 static judgment data, 1 fast movement judgment data, and 1 area judgment data.

[0114] Please refer to Figure 5 , in some embodiments, the motion analysis data includes: motion parameter distribution data, motion trajectory data, motion position data, and kinematic parameter inter-group data, and the behavior analysis data includes: action clustering analysis data, inter-group action distribution data, action distribution statistical data, and action spectrum; the motion behavior analysis unit 220 includes: a motion analysis subunit 221 and a behavior analysis subunit 222;

[0115] The motion analysis subunit 221 is used for:

[0116] Performing parameter distribution analysis on the target object based on the body point motion data to obtain motion parameter distribution data;

[0117] Performing motion trajectory analysis on the target object based on the body point motion data to obtain motion trajectory data;

[0118] Performing motion position analysis on the target object based on the body point motion data to obtain motion position data;

[0119] Performing kinematic parameter inter-group analysis on the target object based on the body point motion data and the preset grouping of the target object to obtain kinematic parameter inter-group data;

[0120] The behavior analysis subunit 222 is used for:

[0121] Performing clustering analysis on the action data to obtain action clustering analysis data;

[0122] Performing inter-group action distribution analysis on the target object based on the action data to obtain inter-group action distribution data;

[0123] Performing action distribution statistical analysis on the target object based on the action data to obtain action distribution statistical data;

[0124] Constructing an action spectrum according to the action data.

[0125] In some embodiments, the motion analysis subunit 221 is capable of performing parameter distribution analysis on a target object based on body point motion data to obtain motion parameter distribution data; performing motion trajectory analysis on the target object based on body point motion data to obtain motion trajectory data; performing motion position analysis on the target object based on body point motion data to obtain motion position data; and performing between-group analysis of kinematic parameters on the target object based on body point motion data and a preset grouping of target objects to obtain between-group kinematic parameter data. Specifically, first, clustering processing is performed on the body point motion data of each sample according to the video time sequence to obtain body point motion clustering data; then, dimensionality reduction processing is performed on the body point motion clustering data to obtain body point motion dimensionality-reduced clustering data. Finally, parameter distribution analysis is performed on the target object based on the body point motion dimensionality-reduced clustering data, and motion parameter distribution data can be obtained. Motion trajectory analysis is performed on the target object based on body point motion data to obtain motion trajectory data, where the motion trajectory data includes two-dimensional velocity-trajectory correlation data and three-dimensional velocity-trajectory correlation data. Motion position analysis is performed on the target object based on body point motion data, and motion position data can be obtained. Between-group analysis of kinematic parameters is performed on the target object based on body point motion data and a preset grouping of target objects, and between-group kinematic parameter data can be obtained, where the method for between-group analysis of kinematic parameters is the Welch independent samples t-test method, and the between-group kinematic parameter data includes between-group kinematic parameter statistical data. Visualization is respectively performed on the motion parameter distribution data, two-dimensional velocity-trajectory correlation data, three-dimensional velocity-trajectory correlation data, motion position data, and between-group kinematic parameter statistical data, and a dimensionality-reduced clustering distribution diagram of kinematic parameters, a two-dimensional velocity-trajectory diagram, a three-dimensional velocity-trajectory diagram, a position heat map, and a between-group kinematic parameter statistical diagram can be obtained; among them, the motion parameter distribution data corresponds to the dimensionality-reduced clustering distribution diagram of kinematic parameters, the two-dimensional velocity-trajectory correlation data corresponds to the two-dimensional velocity-trajectory diagram, the three-dimensional velocity-trajectory correlation data corresponds to the three-dimensional velocity-trajectory diagram, the motion position data corresponds to the position heat map, and the between-group kinematic parameter statistical data corresponds to the between-group kinematic parameter statistical diagram.

[0126] In some embodiments, the behavior analysis subunit 222 can perform clustering analysis on the action data to obtain action clustering analysis data; perform inter-group action distribution analysis on the target object based on the action data to obtain inter-group action distribution data; perform action distribution statistical analysis on the target object based on the action data to obtain action distribution statistical data; and construct an action spectrum according to the action data. Specifically, perform dimensionality reduction processing on the action data by proportion to obtain three-dimensional action proportion data; perform clustering analysis on the three-dimensional action proportion data to obtain action clustering analysis data; perform inter-group action distribution analysis on the target object according to the three-dimensional action proportion data and the preset grouping of the target object to obtain inter-group action distribution analysis data; perform action distribution statistical analysis on all target objects based on the action data to obtain action distribution statistical data, and construct an action spectrum according to the action data. Visualize the action clustering analysis data, the inter-group action distribution analysis data, and the action distribution statistical data respectively to obtain the corresponding three-dimensional action space clustering diagram, the inter-group action distribution dimensionality reduction diagram, and the action distribution statistical result diagram, where the three-dimensional action space clustering diagram corresponds to the action clustering analysis data, the inter-group action distribution dimensionality reduction diagram corresponds to the inter-group action distribution analysis data, and the action distribution statistical result diagram corresponds to the action distribution statistical data.

[0127] In some embodiments, the physiological state change analysis unit 230 can extract physiological data of the target object based on the sensor data to obtain physiological data; then, perform physiological state change analysis on the target object based on the physiological data to obtain physiological state change analysis data. Specifically, the sensor data includes the body temperature data, position data, weight data, humidity data, environmental temperature data, air pressure data, water consumption data, and food consumption data of the target object. The physiological data includes: the feeding and drinking volume and frequency of the mouse at different time periods of each day, the intake frequency of different foods by the mouse, the daily sleep duration and sleep time period of the mouse, the daily weight change, the oxygen consumption, and the carbon dioxide generation. The physiological state change analysis data includes: feeding and drinking frequency data, feeding and drinking line graph data, feeding and drinking preference data, sleep time and duration data, weight change data, exercise volume data, and metabolism data. In addition, the physiological state change analysis unit performs visualization processing on the physiological state change analysis data to obtain a feeding and drinking frequency graph, a feeding and drinking line graph, a feeding and drinking preference graph, a sleep time and duration graph, a weight change curve graph, an exercise volume curve graph, and a metabolism curve graph.

[0128] The data processing module 200 shown in the embodiments of the present application splits the monitored video data through the video analysis unit 210 to obtain a plurality of monitored video frame images; and inputs the plurality of monitored video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data; wherein the motion pose estimation data represents the motion data on each body point of the target object. The motion behavior analysis unit 220 performs motion behavior analysis based on the motion pose estimation data to obtain motion behavior analysis data. The physiological state change analysis unit 230 extracts physiological data of the target object based on the sensor data to obtain physiological data, and performs physiological state change analysis on the target object based on the physiological data to obtain physiological state change analysis data. Therefore, the data processing module 200 shown in the embodiments of the present application can analyze the monitored video data through machine learning and artificial intelligence algorithms, can adapt to the diversity and complexity of the behavior of the target object, and can obtain more accurate motion behavior analysis data. In addition, the data processing module 200 shown in the embodiments of the present application can analyze multi-modal data to obtain motion behavior analysis data and physiological state change analysis data, enabling the object monitoring system to comprehensively analyze multi-faceted data of the target object, which helps to comprehensively understand the overall health status and behavioral responses of the target object.

[0129] In some embodiments, the display module 300 is electrically connected to the data processing module 200, and is configured to obtain the motion behavior analysis data and the physiological state change analysis data in the data processing module 200, and display the motion behavior analysis data and the physiological state change analysis data. Specifically, the display module 300 displays the motion behavior analysis data and the physiological state change analysis data in the form of tables and pictures. Therefore, the display module 300 shown in the embodiments of the present application can display data in a variety of different forms, facilitating the user to view and understand the health status and behavioral status of the target object reflected by the data.

[0130] Please refer to Figure 6 , in some embodiments, the object monitoring system further includes: an operation platform 400, a data storage unit 500, and a data export unit 600;

[0131] The operation platform 400 is communicatively connected to the multi-modal data acquisition module 100 and is configured to set operation parameters and perform experiment control; wherein the operation parameters include: target object information, target data acquisition information;

[0132] The data storage unit 500 is electrically connected to the data processing module 200 and the operation platform 400, and is configured to store the motion behavior analysis data and the physiological state change analysis data;

[0133] The data export unit 600 is electrically connected to the data storage unit 500 and is used to export the motion behavior analysis data and the physiological state change analysis data.

[0134] In some embodiments, the operation platform 400 is communicatively connected to the multi-modal data acquisition module 100 and is used to set operation parameters and conduct experiment control. Specifically, the operation parameters include: target object information and target data acquisition information. The experiment control by the operation platform 400 includes but is not limited to the following steps: selecting monitoring parameters; controlling the start or stop of the experiment; obtaining the working state information of the sensor data acquisition device and the real-time monitoring data of the video data acquisition device; and recording experiment abnormal events. Among them, the working state information of the sensor data acquisition device includes: the real-time working condition information of the sensor data acquisition device, the real-time readings of the sensor data acquisition device, and the record of experiment abnormal events.

[0135] In some embodiments, the data storage unit 500 is electrically connected to the data processing module 200 and the operation platform 400 and is used to store the motion behavior analysis data and the physiological state change analysis data. Specifically, the data storage unit 500 names each experiment's data storage folder with the experiment name and date, and stores the motion behavior analysis data and the physiological state change analysis data in the data storage folder. The motion behavior analysis data and the physiological state change analysis data are stored in the form of briefing and PowerPoint. In addition, the data storage unit 500 can also save the target information configuration file, the real-time readings file of the sensor data acquisition device, the test log record file, and the test video file in each experiment's data storage folder. Among them, the target information configuration file stores: the target object information and the target data acquisition information, the sensor real-time readings file backs up: the sensor data, and the test video file backs up: the monitoring video data.

[0136] As Figure 7 shown, the embodiment of the present application also provides an object monitoring method, which is applied to the above object monitoring system. The method includes but is not limited to steps S701 to S704:

[0137] Step S701, obtaining target multi-modal data; wherein, the target multi-modal data is the data collected by the multi-modal data acquisition module during the motion process and life process of the target object, and the target multi-modal data includes: monitoring video data and sensor data;

[0138] Step S702, performing motion behavior analysis on the target object based on the monitoring video data to obtain motion behavior analysis data; wherein, the monitoring video data is the video data characterizing the motion process of the target object;

[0139] Step S703: Analyze the physiological state changes of the target object based on the sensor data to obtain physiological state change analysis data. Among them, the sensor data represents the physiological data of the target object during the life process.

[0140] Step S704: Send the motion behavior analysis data and the physiological state change analysis data to the display module so that the display module can display the motion behavior analysis data and the physiological state change analysis data.

[0141] The object monitoring method shown in the embodiments of the present application obtains target multimodal data through steps S701 to S704. Among them, the target multimodal data is the data collected by the multimodal data acquisition module during the motion process and life process of the target object, and the target multimodal data includes: monitoring video data and sensor data. Then, based on the monitoring video data, perform motion behavior analysis on the target object to obtain motion behavior analysis data. Among them, the monitoring video data represents the video data of the target object during the motion process; and based on the sensor data, perform physiological state change analysis on the target object to obtain physiological state change analysis data. Among them, the sensor data represents the physiological data of the target object during the life process. Finally, send the motion behavior analysis data and the physiological state change analysis data to the display module so that the display module can display the motion behavior analysis data and the physiological state change analysis data. Therefore, the object monitoring method proposed in the present application can collect the target multimodal data of the target object and analyze the collected target multimodal data, and can obtain both motion behavior analysis data and physiological state change analysis data at the same time, improving the comprehensiveness of data collection and analysis of the object monitoring system, and reducing the time required for multiple replacements of the object monitoring system for data collection and analysis, thereby reducing the time cost of data collection and analysis.

[0142] As Figure 8 shown, in some embodiments, step S702 may include but is not limited to steps S801 to S803:

[0143] Step S801: Perform splitting processing on the monitoring video data to obtain a plurality of monitoring video frame images.

[0144] Step S802: Input the plurality of monitoring video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data. Among them, the motion pose estimation data represents the motion data at each body point of the target object.

[0145] Step S803: Perform motion behavior analysis based on the motion pose estimation data to obtain motion behavior analysis data.

[0146] Steps S801 to S803 shown in the embodiments of the present application first perform a splitting process on the monitored video data to obtain a plurality of monitored video frame images, and then input the plurality of monitored video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data; wherein, the motion pose estimation data represents the motion data at each body point of the target object; finally, motion behavior analysis is performed based on the motion pose estimation data to obtain motion behavior analysis data. Therefore, the present application performs motion pose estimation on a plurality of monitored video frame images based on the pose estimation model to obtain motion pose estimation data, which can adapt to the diversity and complexity of object behaviors to obtain more accurate motion pose estimation data. This makes the motion behavior analysis data obtained based on the motion pose estimation data more accurate.

[0147] As Figure 9 shown, in some embodiments, the motion pose estimation data includes: body point motion data and action data, the motion behavior analysis data includes: motion analysis data and behavior analysis data, the motion analysis data includes: motion parameter distribution data, motion trajectory data, motion position data, and kinematic parameter inter-group data, and the behavior analysis data includes: action clustering analysis data, inter-group action distribution data, action distribution statistical data, and action spectrum; step S803 includes but is not limited to steps S901 to S908:

[0148] Step S901, perform parameter distribution analysis on the target object based on the body point motion data to obtain motion parameter distribution data;

[0149] Step S902, perform motion trajectory analysis on the target object based on the body point motion data to obtain motion trajectory data;

[0150] Step S903, perform motion position analysis on the target object based on the body point motion data to obtain motion position data;

[0151] Step S904, perform kinematic parameter inter-group analysis on the target object based on the body point motion data and a preset grouping of the target object to obtain kinematic parameter inter-group data;

[0152] Step S905, perform clustering analysis on the action data to obtain action clustering analysis data;

[0153] Step S906, perform inter-group action distribution analysis on the target object based on the action data to obtain inter-group action distribution data;

[0154] Step S907, perform action distribution statistical analysis on the target object based on the action data to obtain action distribution statistical data;

[0155] Step S908, construct an action spectrum according to the action data.

[0156] In step S901 of some embodiments, by performing parameter distribution analysis on the target object based on the body point motion data, motion parameter distribution data can be obtained. Specifically, first, the body point motion data of each sample is clustered according to the video time sequence to obtain body point motion clustering data; then, dimensionality reduction processing is performed on the body point motion clustering data to obtain body point motion dimensionality reduction clustering data. Finally, by performing parameter distribution analysis on the target object based on the body point motion dimensionality reduction clustering data, motion parameter distribution data can be obtained.

[0157] In step S902 of some embodiments, by performing motion trajectory analysis on the target object based on the body point motion data, motion trajectory data can be obtained. Specifically, by performing motion trajectory analysis on the target object based on the body point motion data, motion trajectory data is obtained, where the motion trajectory data includes: two-dimensional velocity-trajectory correlation data and three-dimensional velocity-trajectory correlation data.

[0158] In step S904 of some embodiments, by performing between-group analysis of kinematic parameters on the target object based on the body point motion data and a preset grouping of the target object, between-group kinematic parameter data can be obtained. Specifically, the method for between-group analysis of kinematic parameters includes: Welch's independent samples t-test, and the between-group kinematic parameter data includes: between-group statistical data of kinematic parameters.

[0159] After step S904 of some embodiments, step S803 further includes: visualizing the motion parameter distribution data, two-dimensional velocity-trajectory correlation data, three-dimensional velocity-trajectory correlation data, motion position data, and between-group statistical data of kinematic parameters respectively, to obtain a dimensionality reduction clustering distribution map of kinematic parameters, a two-dimensional velocity-trajectory map, a three-dimensional velocity-trajectory map, a position heat map, and a between-group statistical chart of kinematic parameters; where the motion parameter distribution data corresponds to the dimensionality reduction clustering distribution map of kinematic parameters, the two-dimensional velocity-trajectory correlation data corresponds to the two-dimensional velocity-trajectory map, the three-dimensional velocity-trajectory correlation data corresponds to the three-dimensional velocity-trajectory map, the motion position data corresponds to the position heat map, and the between-group statistical data of kinematic parameters corresponds to the between-group statistical chart of kinematic parameters.

[0160] In step S905 of some embodiments, by performing clustering analysis on the action data, action clustering analysis data can be obtained. Specifically, first, dimensionality reduction processing of the proportion is performed on the action data to obtain three-dimensional action proportion data. Then, clustering analysis is performed on the three-dimensional action proportion data to obtain action clustering analysis data.

[0161] After step S908 in some embodiments, step S803 further includes: visualizing the action clustering analysis data, the inter-group action distribution analysis data, and the action distribution statistical data respectively to obtain a corresponding three-dimensional action space clustering graph, an inter-group action distribution dimensionality reduction graph, and an action distribution statistical result graph, where the three-dimensional action space clustering graph corresponds to the action clustering analysis data, the inter-group action distribution dimensionality reduction graph corresponds to the inter-group action distribution analysis data, and the action distribution statistical result graph corresponds to the action distribution statistical data.

[0162] As Figure 10 shown, in some embodiments, step S703 includes but is not limited to steps S1001 to S1002:

[0163] Step S1001, extracting physiological data from the target object based on the sensor data to obtain physiological data;

[0164] Step S1002, analyzing the physiological state change of the target object based on the physiological data to obtain physiological state change analysis data.

[0165] Steps S1001 to S1002 illustrated in the embodiments of the present application extract physiological data from the target object based on the sensor data to obtain physiological data, and then analyze the physiological state change of the target object based on the physiological data to obtain physiological state change analysis data. Therefore, the object monitoring method illustrated in the embodiments of the present application can deeply analyze the physiological data to obtain physiological state change analysis data, and the physiological state change analysis data can more accurately reflect the physiological state change of the target object, facilitating the understanding of the overall health status of the target object.

[0166] Figure 11 is another optional flowchart of the object monitoring method provided by the embodiments of the present application. As Figure 11 shown, before step S701, the object monitoring method further includes but is not limited to steps S1101 to S1103:

[0167] Step S1101, obtaining preset target object information and target data acquisition information; where the target object information is used to represent the basic information of the target object, and the target data acquisition information is used to represent the information of the data to be collected;

[0168] Step S1102, collecting data from the target object according to the preset data collection rules, target object information, and target data acquisition information to obtain candidate multimodal data;

[0169] Step S1103, screening the candidate multimodal data according to the preset information transmission rules to obtain target multimodal data.

[0170] In step S1102 of some embodiments, candidate multi-modal data is filtered according to a preset information transmission rule to obtain target multi-modal data. Specifically, the data acquisition rule is used to characterize whether the target object information and the target data acquisition information of the target object change during the data acquisition process, and whether there are abnormalities in the working state information of the sensor data acquisition device and the video data acquisition device. The working state information of the sensor data acquisition device is used to characterize the working state of the sensor data acquisition device, and the working state information of the video data acquisition device is used to characterize the working state of the video data acquisition device. The video data acquisition device and the sensor data acquisition device are controlled according to the preset data acquisition rule, the target object information, and the target data acquisition information to acquire data from the target object, and candidate multi-modal data is obtained, including: First, the current object information, the current data acquisition information, the working state information of the sensor data acquisition device, and the working state information of the video data acquisition device are acquired; wherein, the current object information is used to characterize the current target object information, and the current data acquisition information is used to characterize the current target data acquisition information. Then, the current target object information is compared with the target object information according to the data acquisition rule, and the current target data acquisition information is compared with the target data acquisition information to obtain a comparison result. If the comparison result indicates that the current target object information is inconsistent with the target object information, the target object information is updated according to the current target object information to obtain updated target object information; if the comparison result indicates that the current target data acquisition information is inconsistent with the target data acquisition information, the target data acquisition information is updated according to the current target data acquisition information to obtain updated target data acquisition information. The above steps of comparing and updating the target object information or the target data acquisition information are repeated until the comparison result indicates that the current target object information is consistent with the target object information and the current target data acquisition information is consistent with the target data acquisition information. If the comparison result indicates that the current target object information is consistent with the target object information and the current target data acquisition information is consistent with the target data acquisition information, one or more are selected from the sensor data acquisition device and the video data acquisition device according to the target object information and the target data acquisition information to obtain a target data acquisition device. Thereafter, the working state of the target data acquisition device is determined according to the target data acquisition device, the working state information of the sensor data acquisition device, and the working state information of the video data acquisition device to obtain the working state of the target data acquisition device; wherein, the working state of the target data acquisition device includes: working state, stopped working state, abnormal state.If the working state of the target data acquisition device indicates that any one of the target data acquisition devices is in a stopped working state, control the target data acquisition device in the stopped working state to change the working state of the target data acquisition device to the working state. If the working state of the target data acquisition device indicates that any one of the target data acquisition devices is in an abnormal state, record the target data acquisition device in the abnormal state, i.e., the abnormal state information, to obtain the experimental abnormal event record information. Stop data acquisition according to the experimental abnormal event record information. If the working state of the target data acquisition device indicates that the target data acquisition device is in the working state, control the target data acquisition device to perform data acquisition on the target object to obtain candidate multimodal data.

[0171] Steps S1101 to S1102 illustrated in the embodiments of the present application first obtain preset target object information and target data acquisition information. Among them, the target object information is used to represent the basic information of the target object, and the target data acquisition information is used to represent the information of the data to be acquired. Then, data acquisition is performed on the target object according to the preset data acquisition rules, target object information, and target data acquisition information to obtain candidate multimodal data. Finally, the candidate multimodal data is screened according to the preset information transmission rules to obtain the target multimodal data. Therefore, the object monitoring method illustrated in the embodiments of the present application can automatically adjust the acquired multimodal data according to real-time data through the preset data acquisition rules to adapt to the changes in the behavior of the target object and the changes in experimental conditions.

[0172] Such as Figure 12As shown in the figure, the object detection method illustrated in the embodiments of the present application includes the following steps: First, install a hardware system (i.e., an object monitoring system) according to preset target object information and target data acquisition information. Then, set operation parameters through an experimental monitoring software (i.e., an operation platform), and at the same time perform a data saving operation, where the data is the operation parameters. After that, perform an experimental control operation according to the operation parameters, and an intelligent phenotype data acquisition unit (i.e., a modality data acquisition module) performs multi-modal data acquisition to obtain multi-modal data, where the multi-modal data includes monitoring video data and sensor data. Input the monitoring video data into an image annotation and model training software (i.e., a video analysis unit), and perform motion pose estimation according to the monitoring video data and a pose estimation model to obtain motion pose estimation data; where the motion pose estimation data represents the motion data on each body point of the target object; then input the motion pose estimation data and the sensor data into a data analysis software (i.e., a motion behavior analysis unit and a physiological state change analysis unit). The data analysis software respectively performs motion behavior analysis on the motion pose estimation data to obtain motion behavior analysis data; extracts physiological data from the target object based on the sensor data to obtain physiological data, and performs physiological state change analysis on the target object based on the physiological data to obtain physiological state change analysis data. Obtain a visual experimental result according to the motion behavior analysis data and the physiological state change analysis data.

[0173] The specific implementation manner of this object monitoring method is basically the same as the specific embodiments of the above object monitoring device, and the embodiments of other steps will not be elaborated here.

[0174] The object monitoring method provided by the embodiments of the present application relates to the field of artificial intelligence technology. The object monitoring method provided by the embodiments of the present application can be applied to a terminal, or can be applied to a server side, or can also be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the object monitoring method, etc., but is not limited to the above forms.

[0175] This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are executed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0176] An embodiment of this application also provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above object monitoring method is implemented. The electronic device can be any intelligent terminal including a tablet computer, in-vehicle computer, etc.

[0177] Please refer to Figure 13 , Figure 13 , which schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0178] A processor 1301, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), microprocessor, Application Specific Integrated Circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of this application;

[0179] A memory 1302, which can be implemented in forms such as Read Only Memory (ROM), static storage devices, dynamic storage devices, or Random Access Memory (RAM). The memory 1302 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1302 and are called by the processor 1301 to execute the object monitoring method of the embodiments of this application;

[0180] An input / output interface 1303, which is used to implement information input and output;

[0181] A communication interface 1304 for implementing communication and interaction between this device and other devices, which can achieve communication through a wired manner (such as USB, network cable, etc.) or through a wireless manner (such as mobile network, WI-FI, Bluetooth, etc.);

[0182] A bus 1305 for transmitting information between various components of the device (such as a processor 1301, a memory 1302, an input / output interface 1303, and a communication interface 1304);

[0183] Among them, the processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304 achieve communication connections with each other inside the device through the bus 1305.

[0184] The embodiment of this application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above object monitoring method is implemented.

[0185] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] The object monitoring system, object monitoring method, electronic device, and storage medium provided by the embodiments of the present application first collect target multi-modal data of a target object through a multi-modal data acquisition module; among them, the target multi-modal data includes: surveillance video data and sensor data, the surveillance video data represents video data of the movement process of the target object, and the sensor data represents physiological data of the life process of the target object. Then, the multi-modal data acquisition module and the data processing module are connected through a communication connection, and the target multi-modal data is transmitted to the data processing module. The data processing module performs motion behavior analysis on the target object based on the surveillance video data to obtain motion behavior analysis data, and performs physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data. Finally, the data processing module and the display module are electrically connected, and the motion behavior analysis data and the physiological state change analysis data are transmitted to the display module, and the motion behavior analysis data and the physiological state change analysis data are displayed through the display module. Therefore, the object monitoring system proposed by the present application can collect the target multi-modal data of the target object and analyze the collected target multi-modal data, and can simultaneously obtain the motion behavior analysis data and the physiological state change analysis data, improving the comprehensiveness of data collection and analysis of the object monitoring system, and reducing the time required for collecting and analyzing data by replacing the object monitoring system multiple times, thereby reducing the time cost of data collection and analysis.

[0187] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation to the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0188] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and may include more or fewer steps than shown in the figures, or combine some steps, or different steps.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0191] As used in the specification of this application and the above-mentioned drawings, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0192] It should be understood that in this application, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0193] In several embodiments provided in this application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the above-mentioned unit division is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0194] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0196] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0197] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, which does not limit the scope of rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall fall within the scope of rights of the embodiments of this application.

Claims

1. Object monitoring system, characterized in that, The system includes: A multi-modal data acquisition module, which is used to acquire target multi-modal data of a target object; wherein, the target multi-modal data includes: surveillance video data and sensor data; A data processing module, which is communicatively connected to the multi-modal data acquisition module, and the data processing module is used for: Performing motion behavior analysis on the target object based on the surveillance video data to obtain motion behavior analysis data; wherein, the surveillance video data represents video data of the motion process of the target object; Performing physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data; wherein, the sensor data represents physiological data of the life process of the target object; A display module, which is electrically connected to the data processing module and is used to display the motion behavior analysis data and the physiological state change analysis data.

2. The object monitoring system according to claim 1, characterized in that, The data processing module includes: a video analysis unit, a motion behavior analysis unit, and a physiological state change analysis unit; The video analysis unit is used for: Performing splitting processing on the surveillance video data to obtain a plurality of surveillance video frame images; Inputting the plurality of surveillance video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data; wherein, the motion pose estimation data represents motion data at each body point of the target object; The motion behavior analysis unit is used for: Performing motion behavior analysis based on the motion pose estimation data to obtain the motion behavior analysis data; The physiological state change analysis unit is used for: Extracting physiological data of the target object based on the sensor data to obtain physiological data; Performing physiological state change analysis on the target object based on the physiological data to obtain the physiological state change analysis data.

3. The object monitoring system according to claim 2, characterized in that, The motion pose estimation data includes: body point motion data and action data, the motion behavior analysis data includes: motion analysis data and behavior analysis data, the motion analysis data includes: motion parameter distribution data, motion trajectory data, motion position data, and kinematic parameter inter-group data, the behavior analysis data includes: action clustering analysis data, inter-group action distribution data, action distribution statistical data, and action spectrum; the motion behavior analysis unit includes: a motion analysis subunit and a behavior analysis subunit; The motion analysis subunit is used for: Performing parameter distribution analysis on the target object based on the body point motion data to obtain the motion parameter distribution data; Performing motion trajectory analysis on the target object based on the body point motion data to obtain the motion trajectory data; Performing motion position analysis on the target object based on the body point motion data to obtain the motion position data; Performing kinematic parameter inter-group analysis on the target object based on the body point motion data and a preset grouping of the target object to obtain the kinematic parameter inter-group data; The behavior analysis subunit is used for: Performing clustering analysis on the action data to obtain the action clustering analysis data; Perform inter-group action distribution analysis on the target object based on the action data to obtain the inter-group action distribution data; Perform action distribution statistical analysis on the target object based on the action data to obtain the action distribution statistical data; Construct an action spectrum according to the action data.

4. The object monitoring system according to claim 1, characterized in that, The multi-modal data acquisition module includes: a data acquisition control device, a video data acquisition device, and a sensor data acquisition device; The data acquisition control device is used for: Obtain preset target object information and target data acquisition information; wherein, the target object information is used to represent the basic information of the target object, and the target data acquisition information is used to represent the information of the data to be acquired; Control the video data acquisition device and the sensor data acquisition device to perform data acquisition on the target object according to preset data acquisition rules, the target object information, and the target data acquisition information to obtain candidate multi-modal data; Screen the candidate multi-modal data according to preset information transmission rules to obtain the target multi-modal data.

5. Object monitoring method, characterized in that, The method is applied to the object monitoring system according to any one of claims 1 to 4, and the method includes: Obtain target multi-modal data; wherein, the target multi-modal data is data collected by a multi-modal data acquisition module during the movement process and daily life process of the target object, and the target multi-modal data includes: surveillance video data and sensor data; Perform motion behavior analysis on the target object based on the surveillance video data to obtain motion behavior analysis data; wherein, the surveillance video data represents the video data of the movement process of the target object; Perform physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data; wherein, the sensor data represents the physiological data of the daily life process of the target object; Send the motion behavior analysis data and the physiological state change analysis data to a display module so that the display module displays the motion behavior analysis data and the physiological state change analysis data.

6. The object monitoring method according to claim 5, characterized in that, The performing motion behavior analysis on the target object based on the surveillance video data to obtain motion behavior analysis data includes: Perform splitting processing on the surveillance video data to obtain a plurality of surveillance video frame images; Input the plurality of surveillance video frame images into a preset pose estimation model for motion pose estimation to obtain motion pose estimation data; wherein, the motion pose estimation data represents the motion data at each body point of the target object; Perform motion behavior analysis based on the motion pose estimation data to obtain the motion behavior analysis data.

7. The object monitoring method according to claim 6, characterized in that, The motion posture estimation data includes: body point motion data and action data. The motion behavior analysis data includes: motion analysis data and behavior analysis data. The motion analysis data includes: motion parameter distribution data, motion trajectory data, motion position data, and kinematic parameter inter-group data. The behavior analysis data includes: action clustering analysis data, inter-group action distribution data, action distribution statistical data, and action spectrum. Performing motion behavior analysis based on the motion posture estimation data to obtain the motion behavior analysis data includes: Performing parameter distribution analysis on the target object based on the body point motion data to obtain the motion parameter distribution data; Performing motion trajectory analysis on the target object based on the body point motion data to obtain the motion trajectory data; Performing motion position analysis on the target object based on the body point motion data to obtain the motion position data; Performing kinematic parameter inter-group analysis on the target object based on the body point motion data and a preset grouping of the target object to obtain the kinematic parameter inter-group data; Performing clustering analysis on the action data to obtain the action clustering analysis data; Performing inter-group action distribution analysis on the target object based on the action data to obtain the inter-group action distribution data; Performing action distribution statistical analysis on the target object based on the action data to obtain the action distribution statistical data; Constructing an action spectrum according to the action data.

8. The object monitoring method according to claim 5, characterized in that, Performing physiological state change analysis on the target object based on the sensor data to obtain physiological state change analysis data includes: Extracting physiological data from the target object based on the sensor data to obtain physiological data; Performing physiological state change analysis on the target object based on the physiological data to obtain the physiological state change analysis data.

9. The object monitoring method according to claim 5, characterized in that, Before obtaining the target multi-modal data of the target object, the method further includes: Obtaining preset target object information and target data collection information; wherein, the target object information is used to represent the basic information of the target object, and the target data collection information is used to represent the information of the data to be collected; Collecting data from the target object according to preset data collection rules, the target object information, and the target data collection information to obtain candidate multi-modal data; Filtering the candidate multi-modal data according to preset information transmission rules to obtain the target multi-modal data.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the object monitoring method according to any one of claims 5 to 9.

Images