A target detection method, device, equipment and computer-readable storage medium
Through the fusion of image and deep data and the combination of YOLO and LSTM models, automated monitoring is achieved, the problem of low manual monitoring efficiency is solved, monitoring efficiency and flexibility is improved, and multiple target objects can be monitored at the same time and timely warnings are made.
Patent Information
- Application Number
- CN202010819962.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-14
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-08-14
AI Technical Summary
The existing manual monitoring is inefficient, especially when multiple special-need populations are required at the same time, resulting in low monitoring efficiency and high labor costs.
By collecting image data and depth data of preset time periods, fuse it into image depth fusion data, and input a pre-trained object detection model, extract the attribute data of the target object and three-dimensional motion trajectory data, and use YOLO and LSTM models for object detection and Motion trajectory tracking generates three-dimensional motion trajectory data and realizes automated monitoring.
It realizes a flexible monitoring process without continuous monitoring, and can monitor multiple target objects at the same time, reduce labor costs, improve monitoring efficiency, and promptly predict and warning to avoid potential dangers.
Smart Images

Figure CN112036267B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image recognition technology, and in particular to a target detection method, device, equipment and computer-readable storage medium. Background Art
[0002] In some application scenarios, it is necessary to pay attention to the dynamics of the target object in order to identify the needs of the target object. For example, in daily life, there are some special needs groups who cannot take care of themselves and have health problems, such as children, the elderly, and patients. These special needs groups need 24-hour monitoring to avoid adverse consequences due to lack of supervision. For example, children are injured because they are unattended, and patients are unattended because they are sick and no one is there to treat them.
[0003] At present, the supervision of people with special needs is mostly done manually, and the guardians generally need to accompany the wards 24 hours a day, because after the guardians leave, the wards may be in unpredictable danger. However, the 24-hour manual supervision method requires the guardians to stay at the supervision site. If multiple wards need to be supervised at the same time, one-to-one guardians are required, making this manual supervision method inefficient. Summary of the invention
[0004] The main purpose of the embodiments of the present invention is to provide a target detection method, device, equipment and computer-readable storage medium to solve the problem of low efficiency of existing manual monitoring.
[0005] In view of the above technical problems, the embodiments of the present invention are implemented through the following technical solutions:
[0006] An embodiment of the present invention provides a target detection method, including: collecting image data of a detection environment and depth data corresponding to a target object at preset time intervals; sequentially fusing the image data and the depth data collected at the same time in order from earliest to latest acquisition time to obtain image depth fusion data corresponding to the same time; sequentially inputting the fused image depth fusion data into a pre-trained target detection model in order from earliest to latest fusion time, extracting attribute data of the target object from the sequentially input image depth fusion data through the target detection model, and generating three-dimensional motion trajectory data corresponding to the target object based on the sequentially input image depth fusion data.
[0007] Among them, the sequential fusion of the image data and the depth data collected at the same moment to obtain the image depth fusion data corresponding to the same moment includes: combining the image data and the depth data collected at the same moment to form a one-dimensional array of image depth corresponding to the same moment, and using the one-dimensional array of image depth as the image depth fusion data corresponding to the same moment.
[0008] Among them, the target detection model includes: a YOLO model and a long short-term memory LSTM model connected to each other; the target detection model extracts the attribute data of the target object from the sequentially input image depth fusion data, and generates the three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data, including: detecting the target object in the sequentially input image depth fusion data through the YOLO model, extracting the three-dimensional coordinate data of the key points of the target object and the attribute data of the target object; tracking the motion trajectory of the target object detected by the YOLO model in the sequentially input image depth fusion data through the LSTM model to obtain the motion trajectory data of the target object, and generating the three-dimensional motion trajectory data of the target object according to the motion trajectory data of the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data.
[0009] The YOLO model includes: a three-dimensional convolution layer; extracting the three-dimensional coordinate data of the key points of the target object includes: extracting the two-dimensional feature data of the key points of the target object in the image data part of the image depth fusion data through the three-dimensional convolution layer, and extracting the one-dimensional feature data of the key points of the target object in the depth data part of the image depth fusion data; generating the three-dimensional coordinate data of the key points of the target object according to the two-dimensional feature data and the one-dimensional feature data; wherein the spatial dimension of the one-dimensional feature data is different from the spatial dimension of the two-dimensional feature data.
[0010] Among them, the motion trajectory data is the regional position data of the target object in the image depth fusion data; the three-dimensional motion trajectory data includes: multiple frames of three-dimensional motion trajectory images; the three-dimensional motion trajectory data of the target object is generated according to the motion trajectory data of the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data, including: constructing a three-dimensional coordinate space; sequentially acquiring the regional position data and the three-dimensional coordinate data of the key points of the target object in each of the image depth fusion data in order from the first to the last fusion time, and obtaining multiple groups of the regional position data and the three-dimensional coordinate data of the key points; for each group of the regional position data and the three-dimensional coordinate data of the key points, according to the regional position data and the three-dimensional coordinate data of the key points, setting the three-dimensional model corresponding to the target object in the three-dimensional coordinate space to generate a frame of three-dimensional motion trajectory image.
[0011] Wherein, after obtaining the motion trajectory data of the target object, the method further includes: comparing the motion trajectory data of the target object with preset abnormal state data; when the similarity between the motion trajectory data of the target object and the abnormal state data is greater than a preset similarity threshold, executing an abnormal alarm operation corresponding to the abnormal state data.
[0012] Among them, before the image depth fusion data obtained by sequentially inputting the fusion into a pre-trained target detection model, the method also includes: simultaneously collecting image data of the detection environment and depth data corresponding to the target object; fusing the image data and the depth data collected at the same time to obtain sample image depth fusion data and annotating attribute data and area location data for the sample image depth fusion data; performing data enhancement processing based on the sample image depth fusion to obtain multiple enhanced image depth fusion data corresponding to the sample image depth fusion data, and taking each of the enhanced image depth fusion data as a sample image depth fusion data, so as to train the target detection model using all the obtained sample image depth fusion data.
[0013] An embodiment of the present invention also provides a target detection device, including: an acquisition module, used to acquire image data of a detection environment and depth data corresponding to a target object at preset time intervals; a fusion module, used to sequentially fuse the image data and the depth data acquired at the same time in a descending order of acquisition time, to obtain image depth fusion data corresponding to the same time; a detection module, used to sequentially input the image depth fusion data obtained by fusion into a pre-trained target detection model in a descending order of fusion time, extract the attribute data of the target object from the sequentially input image depth fusion data through the target detection model, and generate three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data.
[0014] An embodiment of the present invention further provides a target detection device, which includes a processor and a memory; the processor is used to execute a target detection program stored in the memory to implement any of the target detection methods described above.
[0015] An embodiment of the present invention further provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement any of the target detection methods described above.
[0016] The beneficial effects of the embodiments of the present invention are as follows:
[0017] In an embodiment of the present invention, image data of the detection environment where the target object is located and depth data corresponding to the target object are collected, the image data and the depth data are fused into one data and input into a pre-trained target detection model, and the attribute data and three-dimensional motion trajectory data of the target object output by the target detection model are displayed, so that the user can intuitively see the attribute information of the target object and the motion trajectory of the target object. Through the embodiment of the present invention, the user does not need to stay around the target object all the time, and can check the attribute data and three-dimensional motion trajectory data of the target object output by the target detection model. The monitoring process is more flexible, and multiple monitoring objects can be monitored at the same time, with low labor costs and high monitoring efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0019] Figure 1 is a flow chart of a target detection method according to an embodiment of the present invention;
[0020] Figure 2is a flow chart of processing steps of a target detection model according to an embodiment of the present invention;
[0021] Figure 3 is a flowchart of the processing steps of the LSTM model according to one embodiment of the present invention;
[0022] Figure 4 is a structural diagram of a target detection device according to an embodiment of the present invention;
[0023] Figure 5 is a structural diagram of a target detection device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0025] According to an embodiment of the present invention, a target detection method is provided. Figure 1 FIG. 1 is a flow chart of a target detection method according to an embodiment of the present invention.
[0026] Step S110: collecting image data of the detection environment and depth data corresponding to the target object at every preset time period.
[0027] The length of the preset time period is an empirical value or a value obtained through experiments.
[0028] The detection environment refers to the environment where the target object is located.
[0029] The target object refers to the image of the monitored object, for example, the monitored object is a child, an elderly person, a patient, or of course, the monitored object can also be an animal.
[0030] Specifically, the camera and the depth sensor are called at the same time, so that the camera collects image data of the detection environment and the depth sensor collects depth data corresponding to the target object. The shooting interval of the camera and the sampling interval of the depth sensor are the preset time period.
[0031] Image data refers to the data of a frame of image collected by the camera. Furthermore, the camera's field of view is the detection environment.
[0032] Depth data refers to the depth value of the monitored object in the detection environment with the position of the depth sensor as the starting point. Among them, the depth value of the monitored object in the detection environment is used as the depth value corresponding to the target object. Further, the depth sensor is facing the monitored object or the monitored position of the monitored object. The monitoring position is, for example, a children's bed or a patient's bed. Further, if the depth sensor is facing a movable monitored object, a positioning device can be set on the monitored object, so that the positioning device transmits the position of the target object to the depth sensor in real time; or, an infrared receiver is set on the depth sensor, and an infrared transmitter is set on the monitored object, so that the infrared transmitter sends an infrared signal to the infrared receiver, so that the infrared receiver locates the position of the infrared transmitter according to the received infrared signal, and controls the depth sensor to align with the position of the infrared transmitter.
[0033] Step S120 , sequentially fusing the image data and the depth data collected at the same time in order from earliest to latest collection time, to obtain image depth fusion data corresponding to the same time.
[0034] The interval between two adjacent sampling moments is the above-mentioned preset time period. Each acquisition moment can acquire a set of image data and depth data. The previous acquisition moment is earlier than the later acquisition moment.
[0035] Each set of image data and depth data is fused in order from earliest to latest acquisition time to obtain image depth fusion data corresponding to the set of image data and depth data.
[0036] Specifically, the fusion processing includes: combining the image data and the depth data collected at the same time to form a one-dimensional array of image depth corresponding to the same time, and using the one-dimensional array of image depth as the image depth fusion data corresponding to the same time. The image data is used as the image data part (image data element) in the one-dimensional array of image depth, and the depth data is used as the depth data part (depth data element) in the one-dimensional array of image depth. Furthermore, this embodiment does not limit the order of image data elements and depth data elements in the one-dimensional array of image depth.
[0037] Step S130, in order from the earliest to the latest fusion time, the image depth fusion data obtained by fusion is sequentially input into a pre-trained target detection model, and the attribute data of the target object is extracted through the target detection model to generate the three-dimensional motion trajectory data corresponding to the target object.
[0038] The target detection model is used to extract the attribute data of the target object from the sequentially input image depth fusion data, and generate the three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data.
[0039] The target object's attribute data includes, but is not limited to, the target object's category, name, age, height, skin color, clothing, and number. The target object's category is, for example, children, the elderly, and patients. The number is, for example, the hospitalization number indicated on the wristband.
[0040] Three-dimensional motion trajectory data refers to video data of the target object moving in a three-dimensional coordinate space.
[0041] The three-dimensional motion trajectory data includes: a plurality of frames of three-dimensional motion trajectory images. The plurality of frames of three-dimensional motion trajectory images are played to display the motion trajectory of the target object in the three-dimensional coordinate space.
[0042] Furthermore, the three-dimensional motion trajectory data includes: the three-dimensional motion trajectory data of the target object (monitored object) at the current acquisition moment and the three-dimensional motion trajectory data of the target object at a future moment. The future moment refers to a moment after a preset time length from the current acquisition moment. In other words, by training the target detection model, the target detection model predicts the three-dimensional motion trajectory data of the target object at a future moment based on the sequentially input image depth fusion data.
[0043] The attribute data of the target object output by the target detection model and the three-dimensional motion trajectory data corresponding to the target object are displayed on a preset display device.
[0044] In an embodiment of the present invention, image data of the detection environment where the target object is located and depth data corresponding to the target object are collected, the image data and the depth data are fused into one data and input into a pre-trained target detection model, and the attribute data and three-dimensional motion trajectory data of the target object output by the target detection model are displayed, so that the user can intuitively see the attribute information of the target object and the motion trajectory of the target object. Through the embodiment of the present invention, the user does not need to stay around the target object all the time, and can check the attribute data and three-dimensional motion trajectory data of the target object output by the target detection model. The monitoring process is more flexible, and multiple monitoring objects can be monitored at the same time, with low labor costs and high monitoring efficiency.
[0045] Furthermore, since the state of the monitored object is flexible, the types of monitored objects are diverse (children, the elderly, and various patients), and the movement paths of the monitored objects in the detection environment are random, the target detection model is used to track the trajectory of the target object (monitored object) and generate three-dimensional motion trajectory data, which is of great significance for the automated monitoring of the monitored object.
[0046] Furthermore, since the pre-trained target detection model can predict the three-dimensional motion trajectory data of the target object at a future moment, the user can make a pre-judgment based on the three-dimensional motion trajectory data of the target object at a future moment, make a risk assessment for the monitored object, solve the problems encountered by the monitored object in a timely manner, ensure the safety of the monitored object, and avoid the waste of time, labor and economic costs caused by untimely discovery or handling of problems. For example: based on the image data and depth data collected over a period of time, it is predicted that the target object is about to pull out the infusion needle. At this time, the user can appear beside the target object in time to help or prevent the target object from pulling out the needle.
[0047] In order to make the target detection process of the embodiment of the present invention clearer, the structure and function of the target detection model are further described below.
[0048] The target detection model is a pre-trained model. The target detection model includes: a YOLO (You Only Look Once) model and an LSTM (Long Short-Term Memory) model connected to each other.
[0049] The processing of the target detection model is further described below.
[0050] like Figure 2 , which is a flow chart of the processing steps of a target detection model according to an embodiment of the present invention.
[0051] Step S210, sequentially input the fused image depth fusion data into the YOLO model.
[0052] The YOLO model is used to detect a target object, extract attribute data of the target object, detect key points of the target object and extract three-dimensional coordinate data of the key points in image data of image depth fusion data. The number of key points can be multiple.
[0053] Key points refer to the key positions of the target object, such as eyebrows, corners of eyes, corners of mouth, earlobes, shoulders, elbows, knees, etc. Key points can outline the image area of the target object.
[0054] Step S220, detecting the target object from the sequentially input image depth fusion data through the YOLO model, extracting the three-dimensional coordinate data of the key points of the target object and the attribute data of the target object.
[0055] The three-dimensional coordinate data of the key point refers to the coordinates of the key point in the three-dimensional space.
[0056] In this embodiment, in order to more accurately display the three-dimensional motion data of the target object, the three-dimensional coordinate data of each key point among the multiple key points may be extracted.
[0057] The detected target objects and the key points of the detected target objects are output to the LSTM model through the YOLO model.
[0058] Step S230, using the LSTM model to track the motion trajectory of the target object detected by the YOLO model in the image depth fusion data input sequentially, to obtain the motion trajectory data of the target object, and based on the motion trajectory data of the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data, generate the three-dimensional motion trajectory data of the target object.
[0059] The LSTM model is used to track the motion trajectory of the target object according to the output of the YOLO model and generate three-dimensional motion trajectory data of the target object.
[0060] Motion trajectory tracking is the tracking of the key points of the target object.
[0061] The motion trajectory data includes: the regional position data of the target object in the image depth fusion data. The regional position data may be the regional position of a graph formed by key points of the target object. The graph is similar to the target object.
[0062] The image depth fusion data includes an image data part and a depth data part, and the motion trajectory data is the regional position data of the target object in the image data of the image depth fusion data.
[0063] Since the key points of the target object can reflect the posture of the target object, the motion trajectory data can reflect the state of the monitored object (target object). The state of the monitored object can be determined using the motion trajectory data of one frame of image data or the motion trajectory data of multiple frames of image data. The state includes: health state, behavioral state. For example: through the motion trajectory data of one frame of image data, determine whether the monitored object is within the preset range of the detection environment. For another example: through the motion trajectory data of one frame of image data, determine whether the monitored object is wearing a mask and whether it keeps a distance from others. For another example: through the motion trajectory data of multiple frames of image data, determine whether the monitored object has a normal gait and whether the monitored object has convulsions.
[0064] The target detection model may also include a basic network model, the output of which is connected to the input of the YOLO model, and the output of the YOLO model is connected to the input of the LSTM model. The basic network model is used to preprocess the image data in the image depth fusion data, convert the image data into grayscale data, and reduce the noise in the image data.
[0065] The following is a further description of the process of detecting key points using the YOLO model and extracting the three-dimensional coordinate data of the key points.
[0066] After detecting the image of the target object, the YOLO model identifies key points in the image of the target object, and extracts the three-dimensional coordinate data of each key point for the identified multiple preset types of key points. The multiple preset types of key points include but are not limited to: eyebrow key points, shoulder key points, elbow key points, waist key points, knee key points, and footstep key points.
[0067] Furthermore, the YOLO model includes: a three-dimensional convolution layer. Furthermore, the YOLO model is the fourth version of the YOLO algorithm model (YOLOv4). The two-dimensional convolution layer of the YOLO model is expanded to a three-dimensional convolution layer. The three-dimensional convolution layer includes a convolution layer of three channels. The structures of the convolution layers of the three channels are the same, and the convolution layer of each channel is used to extract feature data of a spatial dimension.
[0068] The two-dimensional feature data of the key points of the target object are extracted from the image data part of the image depth fusion data through the three-dimensional convolution layer, and the one-dimensional feature data of the key points of the target object are extracted from the depth data part of the image depth fusion data; the three-dimensional coordinate data of the key points of the target object are generated according to the two-dimensional feature data and the one-dimensional feature data.
[0069] The spatial dimension of the one-dimensional feature data is different from the spatial dimension of the two-dimensional feature data. For example, the two-dimensional feature data includes the X-axis and Y-axis coordinates corresponding to the key point; the one-dimensional feature data includes the Z-axis coordinate corresponding to the key point; the X-axis, Y-axis and Z-axis coordinates corresponding to the key point are fused to obtain the three-dimensional coordinate data corresponding to the key point.
[0070] The following is a further description of the processing of the LSTM model. Figure 3 , which is a flow chart of the processing steps of the LSTM model according to an embodiment of the present invention.
[0071] Step S310, constructing a three-dimensional coordinate space.
[0072] The three-dimensional coordinate space may be a sky box of a preset viewing angle.
[0073] The preset viewing angle is the same as the viewing angle of the camera that collects the image data.
[0074] Step S320, sequentially acquiring the regional position data and the three-dimensional coordinate data of the key points of the target object in each of the image depth fusion data in the order of fusion time from earliest to latest, and obtaining multiple sets of regional position data and three-dimensional coordinate data of the key points.
[0075] The three-dimensional motion trajectory data includes: multiple frames of three-dimensional motion trajectory images. The three-dimensional motion trajectory image refers to the image of the three-dimensional model corresponding to the target object in the three-dimensional coordinate space.
[0076] The regional position data of the target object may reflect the regional position of the target object in the image data.
[0077] The LSTM model can extract the temporal features of multiple frames of image data from the continuously input multiple image deep fusion data, and determine the regional position of the target object in the image data based on the temporal features. The extracted temporal features are the similarities and differences between the previous frame of image data and the next frame of image data. For example, when a person is walking, the image of the person is coherent, and there are some similarities and differences between the two adjacent frames of image data from near to far (or from far to near). The temporal features are used to reflect this temporal relationship, and can represent the similarities and differences between the two frames of image data in the temporal sequence.
[0078] Step S330, for each set of area position data and three-dimensional coordinate data of key points, a three-dimensional model corresponding to the target object is set in the three-dimensional coordinate space according to the area position data and the three-dimensional coordinate data of the key points to generate a frame of three-dimensional motion trajectory image.
[0079] Each set of region position data and three-dimensional coordinate data of key points is extracted from the same image depth fusion data.
[0080] In this embodiment, Unity3D may be used to generate a three-dimensional motion trajectory image of the target object.
[0081] Specifically, a three-dimensional model of the monitored object, that is, a three-dimensional model corresponding to the target object, can be collected in advance. The regional position data indicates the regional position of the target object in the XOY plane; the three-dimensional coordinate data of the key point indicates the position of the key point of the target object in the three-dimensional coordinate space; the three-dimensional model is set at the regional position of the target object in the XOY plane; the mapping points to which each key point of the target object is mapped on the three-dimensional model are determined, and the three-dimensional coordinates of each mapping point are adjusted to the three-dimensional coordinates of the corresponding key point; after the adjustment is completed, an image of the three-dimensional model under a preset viewing angle is generated as a frame of three-dimensional motion trajectory image corresponding to the target object.
[0082] In this embodiment, since the motion trajectory data of the target object can be used to indicate the status of the monitored object, in order to avoid missing abnormal behavior of the monitored object, after obtaining the motion trajectory data of the target object, the motion trajectory data of the target object is compared with preset abnormal status data; when the similarity between the motion trajectory data of the target object and the abnormal status data is greater than a preset similarity threshold, the abnormal alarm operation corresponding to the abnormal status data is executed.
[0083] Abnormal state data refers to the image data when the monitored object exhibits abnormal behavior.
[0084] The similarity threshold is an empirical value or a value obtained through experiments.
[0085] The abnormal alarm operation includes: issuing a preset alarm tone, and / or sending a message with alarm content corresponding to the abnormal state data to the target user. The message is a text message and / or a voice message.
[0086] For example: the monitored object is a patient, the motion trajectory data of the target object is the patient pulling out the infusion needle, and the abnormal state data is the image data of the monitored object pulling out the infusion needle. In this way, the similarity between the motion trajectory data of the target object and the abnormal state data is greater than the similarity threshold, and a preset alarm sound is issued or information is sent to the nurse station that the patient is trying to pull out the needle.
[0087] In this embodiment, the target detection model is pre-trained. Before training the target detection model, a sample data set needs to be set for the target detection model. The sample data set includes a plurality of deeply fused data of sample images that have been annotated.
[0088] Specifically, the image data of the detection environment and the depth data corresponding to the target object are collected simultaneously; the image data and the depth data collected simultaneously are fused to obtain sample image depth fusion data and the attribute data and regional location data are annotated for the sample image depth fusion data. Furthermore, when collecting the image data of the detection environment, image data of multiple angles such as the side, front, and back of the monitored object are collected as much as possible to ensure the diversity of the sample image depth fusion data, so as to better perform three-dimensional detection of the target object.
[0089] When the amount of sample image depth fusion data is small, the sample image depth fusion data can be enhanced, that is, based on the existing sample image depth fusion data, the amount of sample image depth fusion data is expanded. Specifically, data enhancement processing is performed based on the sample image depth fusion to obtain multiple enhanced image depth fusion data corresponding to the sample image depth fusion data, and each enhanced image depth fusion data is used as a sample image depth fusion data, so as to train the target detection model using all the obtained sample image depth fusion data.
[0090] Furthermore, the image data in the sample image depth fusion data is randomly cropped, scaled, rotated, flipped, mirrored, noise is added, and the contrast, brightness, chroma, etc. are randomly adjusted to obtain new image data, and the new image data is combined with the depth data in the sample image depth fusion data to obtain new image depth fusion data, and the new image depth fusion data is used as the sample image depth fusion data.
[0091] The obtained sample image depth fusion data is stored in a preset sample data set.
[0092] Before training the target detection model, the training data of the target detection model is set. In this embodiment, the training data includes but is not limited to: the number of iterations, the initial value of the weight of the target detection model and the network structure, the learning rate, and the convolution kernel.
[0093] When training the target detection model, the gradient descent method is used to train the target detection model. In the process of training the target detection model using the training data set, the parameters and weights of the target detection model are continuously adjusted according to the preset number of iterations. After the iteration, if the target detection model has not converged, the network structure of the target detection model is adjusted and the target detection model is retrained until the target detection model converges.
[0094] Furthermore, in order to increase the accuracy and robustness of the YOLO model, multi-scale training is added to the YOLO model, the image data in the sample image deep fusion data is converted into multiple scales, and the YOLO model is trained using image data of multiple scales. For example, the image width and height in the configuration file of the YOLO model is set to 640*640 to improve the detection accuracy of the YOLO model for small targets.
[0095] In this embodiment, the accuracy and stability of the target detection model are determined using a loss function. When the loss value of the target detection model is less than a preset loss threshold, it means that the target detection model has converged. Furthermore, the loss function can be a confidence loss function, a classification loss function, or a loss function based on a target rectangular selection box. For example: the distance intersection over union (DIOU) loss function or the complete intersection over union (CIOU) loss function can be used to calculate the loss value of the target detection model, and when the loss value is greater than a preset loss threshold, the parameters of the target detection model are adjusted.
[0096] The embodiment of the present invention also provides a target detection device. Figure 4FIG. 4 is a structural diagram of a target detection device according to an embodiment of the present invention.
[0097] The target detection device includes: a collection module 410 , a fusion module 420 and a detection module 430 .
[0098] The acquisition module 410 is used to acquire image data of the detection environment and depth data corresponding to the target object at every preset time period.
[0099] The fusion module 420 is used to sequentially fuse the image data and the depth data collected at the same time in the order of collection time from earliest to latest, so as to obtain the image depth fusion data corresponding to the same time.
[0100] The detection module 430 is used to input the image depth fusion data obtained by fusion into a pre-trained target detection model in sequence according to the fusion time from the earliest to the latest, extract the attribute data of the target object from the sequentially input image depth fusion data through the target detection model, and generate the three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data.
[0101] The functions of the device described in the embodiment of the present invention have been described in the above method embodiment. Therefore, for details not provided in the description of this embodiment, please refer to the relevant description in the above embodiment and will not be repeated here.
[0102] This embodiment provides a target detection device. Figure 5 FIG. 4 is a structural diagram of a target detection device according to an embodiment of the present invention.
[0103] In this embodiment, the target detection device includes, but is not limited to: a processor 510 and a memory 520 .
[0104] The processor 510 is used to execute the target detection program stored in the memory 520 to implement the above target detection method.
[0105] Specifically, the processor 510 is used to execute the target detection program stored in the memory 520 to implement the following steps: collect image data of the detection environment and depth data corresponding to the target object at every preset time period; sequentially fuse the image data and the depth data collected at the same time in order from the earliest to the latest acquisition time to obtain the image depth fusion data corresponding to the same time; sequentially input the fused image depth fusion data into a pre-trained target detection model in order from the earliest to the latest fusion time, extract the attribute data of the target object from the sequentially input image depth fusion data through the target detection model, and generate three-dimensional motion trajectory data corresponding to the target object based on the sequentially input image depth fusion data.
[0106] Among them, the sequential fusion of the image data and the depth data collected at the same moment to obtain the image depth fusion data corresponding to the same moment includes: combining the image data and the depth data collected at the same moment to form a one-dimensional array of image depth corresponding to the same moment, and using the one-dimensional array of image depth as the image depth fusion data corresponding to the same moment.
[0107] Among them, the target detection model includes: a YOLO model and a long short-term memory LSTM model connected to each other; the target detection model extracts the attribute data of the target object from the sequentially input image depth fusion data, and generates the three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data, including: detecting the target object in the sequentially input image depth fusion data through the YOLO model, extracting the three-dimensional coordinate data of the key points of the target object and the attribute data of the target object; tracking the motion trajectory of the target object detected by the YOLO model in the sequentially input image depth fusion data through the LSTM model to obtain the motion trajectory data of the target object, and generating the three-dimensional motion trajectory data of the target object according to the motion trajectory data of the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data.
[0108] The YOLO model includes: a three-dimensional convolution layer; extracting the three-dimensional coordinate data of the key points of the target object includes: extracting the two-dimensional feature data of the key points of the target object in the image data part of the image depth fusion data through the three-dimensional convolution layer, and extracting the one-dimensional feature data of the key points of the target object in the depth data part of the image depth fusion data; generating the three-dimensional coordinate data of the key points of the target object according to the two-dimensional feature data and the one-dimensional feature data; wherein the spatial dimension of the one-dimensional feature data is different from the spatial dimension of the two-dimensional feature data.
[0109] Among them, the motion trajectory data is the regional position data of the target object in the image depth fusion data; the three-dimensional motion trajectory data includes: multiple frames of three-dimensional motion trajectory images; the three-dimensional motion trajectory data of the target object is generated according to the motion trajectory data of the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data, including: constructing a three-dimensional coordinate space; sequentially acquiring the regional position data and the three-dimensional coordinate data of the key points of the target object in each of the image depth fusion data in order from the first to the last fusion time, and obtaining multiple groups of the regional position data and the three-dimensional coordinate data of the key points; for each group of the regional position data and the three-dimensional coordinate data of the key points, according to the regional position data and the three-dimensional coordinate data of the key points, setting the three-dimensional model corresponding to the target object in the three-dimensional coordinate space to generate a frame of three-dimensional motion trajectory image.
[0110] Wherein, after obtaining the motion trajectory data of the target object, the method further includes: comparing the motion trajectory data of the target object with preset abnormal state data; when the similarity between the motion trajectory data of the target object and the abnormal state data is greater than a preset similarity threshold, executing an abnormal alarm operation corresponding to the abnormal state data.
[0111] Among them, before the image depth fusion data obtained by sequentially inputting the fusion into a pre-trained target detection model, the method also includes: simultaneously collecting image data of the detection environment and depth data corresponding to the target object; fusing the image data and the depth data collected at the same time to obtain sample image depth fusion data and annotating attribute data and area location data for the sample image depth fusion data; performing data enhancement processing based on the sample image depth fusion to obtain multiple enhanced image depth fusion data corresponding to the sample image depth fusion data, and taking each of the enhanced image depth fusion data as a sample image depth fusion data, so as to train the target detection model using all the obtained sample image depth fusion data.
[0112] The embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium here stores one or more programs. The computer-readable storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.
[0113] When one or more programs in the computer-readable storage medium can be executed by one or more processors to implement the above-mentioned target detection method. Since the target detection method has been described in detail above, it will not be repeated here.
[0114] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.
Claims
1. A target detection method, characterized in that: include: Collect image data of the detection environment and depth data corresponding to the target object at every preset time period; In order of acquisition time from earliest to latest, the image data and the depth data acquired at the same time are sequentially fused to obtain image depth fusion data corresponding to the same time; In order from the earliest to the latest fusion time, the image depth fusion data obtained by fusion are sequentially input into a pre-trained target detection model, and the target detection model is used to extract the attribute data of the target object from the sequentially input image depth fusion data, and the three-dimensional motion trajectory data corresponding to the target object is generated according to the sequentially input image depth fusion data; wherein the three-dimensional motion trajectory data includes: the three-dimensional motion trajectory data of the target object at the current acquisition time and the three-dimensional motion trajectory data of the target object at the future time; the target detection model includes: an interconnected YOLO model and a long short-term memory LSTM model; the attribute data of the target object is extracted from the sequentially input image depth fusion data by the target detection model The method comprises the following steps: detecting the target object in the sequentially input image depth fusion data through the YOLO model, extracting the three-dimensional coordinate data of the key points of the target object and the attribute data of the target object; tracking the motion trajectory of the target object detected by the YOLO model in the sequentially input image depth fusion data through the LSTM model to obtain the motion trajectory data of the target object, and generating the three-dimensional motion trajectory data of the target object according to the motion trajectory data of the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data.
2. The method according to claim 1, characterized in that The sequentially fusing the image data and the depth data collected at the same time to obtain the image depth fusion data corresponding to the same time includes: The image data and the depth data collected at the same time are combined to form a one-dimensional array of image depth corresponding to the same time, and the one-dimensional array of image depth is used as the image depth fusion data corresponding to the same time.
3. The method according to claim 1, characterized in that: The YOLO model includes: a three-dimensional convolutional layer; The step of extracting the three-dimensional coordinate data of the key points of the target object comprises: Extracting two-dimensional feature data of key points of the target object in the image data part of the image depth fusion data by the three-dimensional convolution layer, and extracting one-dimensional feature data of key points of the target object in the depth data part of the image depth fusion data; The three-dimensional coordinate data of the key points of the target object are generated according to the two-dimensional feature data and the one-dimensional feature data; wherein the spatial dimension of the one-dimensional feature data is different from the spatial dimension of the two-dimensional feature data.
4. The method according to claim 1, characterized in that: The motion trajectory data is the regional position data of the target object in the image depth fusion data; The three-dimensional motion trajectory data includes: multiple frames of three-dimensional motion trajectory images; The generating of the three-dimensional motion trajectory data of the target object according to the motion trajectory data of the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data includes: Construct a three-dimensional coordinate space; Sequentially acquiring the region position data and the three-dimensional coordinate data of the key points of the target object in each of the image depth fusion data in the order of fusion time from the earliest to the latest, and obtaining multiple sets of the region position data and the three-dimensional coordinate data of the key points; For each set of the area position data and the three-dimensional coordinate data of the key points, a three-dimensional model corresponding to the target object is set in the three-dimensional coordinate space according to the area position data and the three-dimensional coordinate data of the key points to generate a frame of three-dimensional motion trajectory image.
5. The method according to claim 1, characterized in that After obtaining the motion trajectory data of the target object, the method further includes: Comparing the motion trajectory data of the target object with preset abnormal state data; When the similarity between the motion trajectory data of the target object and the abnormal state data is greater than a preset similarity threshold, an abnormal alarm operation corresponding to the abnormal state data is performed.
6. The method according to any one of claims 1 to 5, characterized in that Before sequentially inputting the image depth fusion data obtained by fusion into a pre-trained target detection model, the method further includes: At the same time, image data of the detection environment and depth data corresponding to the target object are collected; Fusing the image data and the depth data collected simultaneously to obtain sample image depth fusion data and marking attribute data and regional position data for the sample image depth fusion data; Data enhancement processing is performed based on the sample image depth fusion to obtain multiple enhanced image depth fusion data corresponding to the sample image depth fusion data, and each enhanced image depth fusion data is used as a sample image depth fusion data, so as to train the target detection model using all the obtained sample image depth fusion data.
7. A target detection device, characterized in that: include: An acquisition module, used to acquire image data of the detection environment and depth data corresponding to the target object at every preset time period; A fusion module, used to sequentially fuse the image data and the depth data collected at the same time in order from the earliest to the latest acquisition time, to obtain image depth fusion data corresponding to the same time; The detection module is used to input the image depth fusion data obtained by fusion into a pre-trained target detection model in sequence according to the order of fusion time from the earliest to the latest, and extract the attribute data of the target object from the sequentially input image depth fusion data through the target detection model, and generate the three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data; wherein the three-dimensional motion trajectory data includes: the three-dimensional motion trajectory data of the target object at the current acquisition time and the three-dimensional motion trajectory data of the target object at the future time; the target detection model includes: an interconnected YOLO model and a long short-term memory LSTM model; the target detection model extracts the target object from the sequentially input image depth fusion data through the target detection model The method comprises the following steps: detecting the target object in the sequentially input image depth fusion data through the YOLO model, extracting the three-dimensional coordinate data of the key points of the target object and the attribute data of the target object; tracking the motion trajectory of the target object detected by the YOLO model in the sequentially input image depth fusion data through the LSTM model to obtain the motion trajectory data of the target object, and generating the three-dimensional motion trajectory data of the target object according to the motion trajectory data of the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data.
8. A target detection device, characterized in that: The target detection device includes a processor and a memory; the processor is used to execute a target detection program stored in the memory to implement the target detection method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the target detection method according to any one of claims 1 to 6.
