A sample generation method, a training method, a recognition method and device
By generating virtual character work images and generating annotation information for them, the problem of feature differences between virtual character training samples and real character training samples is solved, thereby improving the training efficiency and accuracy of the detection model for work risk monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENHUA TIANJIN COAL TERMINAL
- Filing Date
- 2023-08-02
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, deep learning-based job risk monitoring methods require a large amount of data to train the detection model, and there are feature differences between virtual character training samples and real character training samples, which affects the detection effect.
By acquiring images of people and their environment with and without target behaviors, pose information is extracted. Then, a human body generation network and a background fusion network are used to generate virtual human working images and generate annotation information for them. A sample training set is constructed to reduce the difference between virtual human images and real human images.
实现了高效构建数据集，减少了虚拟人体图像与真实人体图像的差异，提高了检测模型的训练效果和识别准确性。
Smart Images
Figure CN117173500B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of sample generation technology, and in particular to a sample generation method, training method, recognition method and apparatus. Background Technology
[0002] In the field of operational risk monitoring, traditional methods rely on managerial inspections and reviewing on-site videos to check for safety risks. While these methods offer some safety management benefits, they also suffer from low efficiency and inaccurate data. With the rapid development of deep learning, intelligent monitoring methods based on deep learning are widely used in operational risk monitoring. However, deep learning-based methods require a large amount of data to train the detection model to achieve ideal detection and recognition results. Therefore, the ability to quickly and easily construct datasets is of significant safety and economic importance for operational risk monitoring. In related technologies, differences exist between the virtual character features obtained from training samples with virtual characters and the real character features obtained from training samples with real characters, affecting the training effect of the detection model. Summary of the Invention
[0003] To address the aforementioned problems, this application provides a sample generation method, a training method, a recognition method, and an apparatus.
[0004] This application provides a sample generation method, the method comprising:
[0005] Acquire the image of the first person exhibiting the target behavior, the image of the second person not exhibiting the target behavior, and the image of the target environment;
[0006] Extract the pose information of the first person image and the second person image;
[0007] The posture information and normally distributed random noise are input into the human body generation network to generate a virtual human body image that conforms to the posture information.
[0008] The virtual human image, the target environment image, and the human coordinate information are input into a background fusion network to generate a virtual character working image.
[0009] Generate annotation information for the virtual character's working image to obtain the virtual character's working image with annotation information;
[0010] A sample training set is generated based on the working images of the virtual character with labeled information.
[0011] In some embodiments, the human body generation network includes a first generator and a first determiner, and the method includes:
[0012] Obtain a first sample image of a person exhibiting the target behavior and a second sample image of a person not exhibiting the target behavior;
[0013] Extract the real human images and human pose information of the first sample human image and the second sample human image;
[0014] The sample human pose information and sample random noise conforming to a normal distribution are input into the first generator to generate a sample virtual human image that conforms to the sample pose information;
[0015] The sample real human body image or the sample virtual human body image is input into the first discriminator, and the first probability is output;
[0016] The weights of the first generator are updated by back gradient calculation based on the comparison results between the sample virtual human image and the corresponding first sample human image or second sample human image, and the weights of the first discriminator are updated by back gradient calculation based on the first probability.
[0017] Repeat the above steps until the human body generation network converges.
[0018] In some embodiments, the background fusion network includes a second generator and a second determiner, and the method includes:
[0019] The sample target environment image, sample real person working image, sample human body coordinate information, and sample virtual human body image are acquired, wherein the sample virtual human body image is generated by the human body generation network;
[0020] The sample target environment image, the sample virtual human image, and the sample human coordinate information are input into the second generator to generate a sample virtual human working image.
[0021] The sample virtual character working image or the sample real character working image is input into the second judge, and the second probability is output;
[0022] The second generator is updated by back gradient calculation based on the comparison results between the sample virtual character working image and the corresponding sample real character working image, and the weights of the second discriminator are updated by back gradient calculation based on the second probability.
[0023] Repeat the above steps until the background fusion network converges.
[0024] In some embodiments, before generating annotation information for the virtual character's working image and obtaining the virtual character's working image with annotation information, the method includes:
[0025] The virtual character working image is preprocessed, including random rotation, translation, brightness variation and / or noise addition.
[0026] In some embodiments, generating annotation information for the virtual character's working image to obtain the virtual character's working image with annotation information includes:
[0027] Obtain the human coordinate information and behavior information of the virtual human image in the virtual character working image, wherein the human coordinate information includes the human coordinate position of the virtual human image in the target environment image and the width and height values of the virtual human image, and the behavior information is the target behavior of the virtual human image;
[0028] Based on the human body coordinate information and behavior information, annotation information is generated for the virtual character working image, resulting in the virtual character working image with annotation information.
[0029] This application provides a training method that uses the aforementioned sample training set, and the detection model includes a human detection model and a behavior recognition model. The method includes:
[0030] The annotation information of all the virtual character working images in the sample training set is statistically analyzed, and the width and height values of the labeled target boxes are obtained based on the annotation information;
[0031] K-Means clustering analysis was performed on the width and height values of the target box to obtain the width and height values of multiple anchor boxes;
[0032] The virtual character working images in the sample training set are input into the human detection model. Based on the width and height values of multiple anchor boxes, the predicted bounding boxes and confidence scores of the human body at different scales are obtained.
[0033] Calculate the predicted bounding box loss value and the confidence loss value, and perform gradient calculation to update the weights of the human detection model based on the predicted bounding box loss value and the confidence loss value;
[0034] Repeat the above training steps until the predicted box loss value and the confidence loss value converge or the specified number of training iterations are reached.
[0035] The working image of the virtual character exhibiting the target behavior is input into the human body detection model, and a human body image is output.
[0036] The human image is input into the behavior recognition model, and a third probability is output.
[0037] Calculate the third loss value of the third probability, and update the weights of the behavior recognition model based on the gradient calculation of the third loss value;
[0038] Repeat the above training steps until the third loss value converges or the specified number of training iterations are reached.
[0039] This application provides a recognition method that applies the detection model described above. The method includes:
[0040] Acquire and decode surveillance video of the target environment to obtain single-frame images;
[0041] Preprocessing of single-frame images includes adaptive image scaling and pixel normalization;
[0042] The preprocessed single-frame image is input into the detection model, which outputs the probability of the presence of the target behavior.
[0043] In some embodiments, after inputting the preprocessed single-frame image into the detection model and outputting the probability of the presence of target behavior, the method includes:
[0044] Compare the probability with a preset threshold;
[0045] If the probability is greater than the preset threshold, an alarm message is generated.
[0046] This application provides a sample generation apparatus, including:
[0047] The first acquisition module is used to acquire the first person image with target behavior, the second person image without target behavior, and the target environment image;
[0048] The extraction module is used to extract the pose information of the first person image and the second person image;
[0049] The first generation module is used to input the posture information and normally distributed random noise into the human body generation network to generate a virtual human body image that conforms to the posture information.
[0050] The second generation module is used to input the virtual human image, the target environment image and human coordinate information into the background fusion network to generate a virtual human working image.
[0051] The annotation module is used to generate annotation information for the virtual character's working image, thereby obtaining the virtual character's working image with annotation information;
[0052] The third generation module is used to generate a sample training set based on the virtual character's working image with labeled information.
[0053] This application provides a training device that uses the aforementioned sample training set. The detection model includes a human detection model and a behavior recognition model, comprising:
[0054] The statistics module is used to collect the annotation information of all the virtual character working images in the sample training set, and obtain the width and height values of the target box for annotation based on the annotation information;
[0055] The clustering analysis module is used to perform K-Means clustering analysis on the width and height values of the target boxes to obtain the width and height values of multiple anchor boxes;
[0056] The first input module is used to input the virtual character working images in the sample training set into the human body detection model, and obtain the prediction boxes and confidence scores of the human body at different scales based on the width and height values of multiple anchor boxes.
[0057] The first calculation module is used to calculate the prediction box loss value and the confidence loss value, and to perform gradient calculation to update the weights of the human detection model based on the prediction box loss value and the confidence loss value.
[0058] The first repetition module is used to repeat the above training steps until the prediction box loss value and the confidence loss value converge or reach the specified number of training times.
[0059] The second input module is used to input the working image of the virtual character with the target behavior into the human body detection model and output a human body image.
[0060] The third input module is used to input the human image into the behavior recognition model and output a third probability that the target behavior exists;
[0061] The second calculation module is used to calculate the third loss value based on the third probability, and to perform gradient calculation to update the weights of the behavior recognition model based on the third loss value.
[0062] The second repetition module is used to repeat the above training steps until the third loss value converges or the specified number of training iterations are reached.
[0063] This application provides an identification device that applies the detection model described above, including:
[0064] The second acquisition module is used to acquire and decode the surveillance video of the target environment to obtain a single frame image;
[0065] The preprocessing module is used to perform adaptive image scaling and pixel normalization preprocessing on single-frame images;
[0066] The fourth input module is used to input the preprocessed single-frame image into the detection model and output the probability of the existence of target behavior.
[0067] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, performs any of the methods described above.
[0068] This application provides a storage medium storing a computer program that can be executed by one or more processors and can be used to implement the method described in any of the above-described embodiments.
[0069] This application provides a sample generation method, training method, recognition method, and apparatus. The method involves acquiring a first person image exhibiting target behavior, a second person image without target behavior, and a target environment image; extracting pose information from the first and second person images; inputting the pose information and normally distributed random noise into a human body generation network to generate a virtual human image conforming to the pose information, thereby making the virtual human image feature diverse and reducing the difference between the virtual and real human images; inputting the virtual human image, target environment image, and human coordinate information into a background fusion network to generate a virtual person working image, further reducing the difference between the virtual and real person working images; generating annotation information for the virtual person working image to obtain a virtual person working image with annotation information; and generating a sample training set based on the virtual person working image with annotation information, avoiding manual annotation of the sample training set. Attached Figure Description
[0070] The present application will be described in more detail below based on embodiments and with reference to the accompanying drawings.
[0071] Figure 1 A schematic flowchart illustrating the sample generation method provided in this application embodiment;
[0072] Figure 2 A schematic diagram illustrating the training process of the human generative network provided in an embodiment of this application;
[0073] Figure 3 This is a schematic diagram of human gene generation network training provided in an embodiment of this application;
[0074] Figure 4 A schematic diagram illustrating the background fusion network training process provided in the embodiments of this application;
[0075] Figure 5 This is a schematic diagram of background fusion network training provided for an embodiment of this application;
[0076] Figure 6 A schematic diagram illustrating the process of generating annotations provided in an embodiment of this application;
[0077] Figure 7 A flowchart illustrating the training method provided in an embodiment of this application;
[0078] Figure 8 A flowchart illustrating the identification method provided in this application embodiment;
[0079] Figure 9 Another schematic diagram of the identification method provided in the embodiments of this application;
[0080] Figure 10 This is a schematic diagram of the sample generation device provided in the embodiments of this application;
[0081] Figure 11 This is a schematic diagram of the structure of the training device provided in the embodiments of this application;
[0082] Figure 12 This is a schematic diagram of the structure of the identification device provided in the embodiments of this application;
[0083] Figure 13 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0084] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0085] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0086] If the application documents contain similar descriptions such as "first, second, third", the following explanation shall be added: In the following description, the terms "first, second, third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
[0087] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0088] To address the problems existing in related technologies, this application provides a sample generation method. This method is applied to an electronic device, such as a mobile phone or computer. The functions implemented by the sample generation method provided in this application can be achieved by the processor of the electronic device calling program code, which can be stored in a computer storage medium.
[0089] This application provides a sample generation method, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating the sample generation method provided in an embodiment of this application. The method includes:
[0090] Step S1: Obtain the image of the first person exhibiting the target behavior, the image of the second person not exhibiting the target behavior, and the image of the target environment;
[0091] In the embodiments of this application, an image of a first person exhibiting the target behavior, an image of a second person not exhibiting the target behavior, and an image of the target environment can be collected via a network. Alternatively, an image of the first person exhibiting the target behavior, an image of the second person not exhibiting the target behavior, and an image of the target environment can be captured by a photographic device. A combination of a network and a photographic device can also be used to acquire these images. The target behavior can be set according to user needs; for example, the target behavior could be smoking or making a phone call. There can be one or more target behaviors, and the embodiments of this application do not limit this.
[0092] Step S2: Extract the pose information of the first person image and the second person image;
[0093] In the embodiments of this application, the pose information of the first person image and the second person image can be extracted using a human image segmentation tool.
[0094] Step S3: Input the posture information and normally distributed random noise into the human body generation network to generate a virtual human body image that conforms to the posture information;
[0095] In the embodiments of this application, it is understood that the normally distributed random noise is the feature information constituting the virtual human body image, other than pose information, such as gender, age, skin color, and clothing color. The normally distributed random noise makes the synthesized virtual human body image feature-rich, and ensures that the virtual human body image does not differ from the real human body image.
[0096] Step S4: Input the virtual human image, the target environment image, and the human coordinate information into the background fusion network to generate a virtual character working image;
[0097] In the embodiments of this application, the target environment image refers to the environmental background, such as a port, gas station, library, etc. The human body coordinate information includes the preset human body coordinate position of the virtual human body image in the target environment image, as well as the width and height of the human body image. This facilitates the fusion of the virtual human body image into the target environment image based on the preset human body coordinate position and the width and height of the human body image, ensuring that the virtual human body image at work is indistinguishable from the real human body image at work.
[0098] Step S5: Generate annotation information for the virtual character working image to obtain the virtual character working image with annotation information;
[0099] In the embodiments of this application, it can be understood that the annotation information refers to the human coordinate information and behavior information corresponding to the virtual human image in the virtual character working image. The human coordinate information includes the human coordinate position of the virtual human image in the target environment image and the width and height of the human image. The behavior information includes whether the virtual human image has a target behavior or not. The existence of a target behavior can indicate which specific target behaviors exist, such as smoking or making a phone call. There can be one or more target behaviors, and the embodiments of this application do not limit this.
[0100] Step S6: Generate a sample training set based on the virtual character working image with labeled information.
[0101] In the embodiments of this application, a sample training set is generated based on the virtual character working image with annotation information, thus avoiding the need for manual annotation of the sample training set.
[0102] In some embodiments, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the training process for a human body generation network provided in an embodiment of this application. The human body generation network includes a first generator and a first judge. The method includes:
[0103] Step S11: Obtain the first sample image of a person exhibiting the target behavior and the second sample image of a person not exhibiting the target behavior;
[0104] Step S12: Extract the real human body image and human body pose information of the first sample human body image and the second sample human body image;
[0105] Step S13: Input the sample human pose information and the sample random noise that conforms to a normal distribution into the first generator to generate a sample virtual human image that conforms to the sample pose information;
[0106] Step S14: Input the real human body image or the virtual human body image of the sample into the first discriminator and output the first probability;
[0107] Step S15: Update the weights of the first generator by back gradient calculation based on the comparison results between the sample virtual human image and the corresponding first sample human image or second sample human image; update the weights of the first discriminator by back gradient calculation based on the first probability.
[0108] Step S16: Repeat the above steps until the human body generation network converges.
[0109] In the embodiments of this application, such as Figure 3 As shown, sample human images I are obtained through the network. M Among them, the image of this person I M The dataset includes a first sample human image exhibiting the target behavior and a second sample human image not exhibiting the target behavior. Sample human images (I) are extracted using a human image segmentation tool. M Sample human images I pr and sample human posture information C p , to obtain the human posture information of the sample C p Using normally distributed sample random noise Z as the first generator G S The input, the first generator G S Generate a sample virtual human image I that matches the pose. pf The corresponding human pose information C will be mapped to the sample. p Sample human body image I pr Or sample virtual human image I pf Input discriminator D s First discriminator D s The output will be the first probability P that belongs to a real human image. s Based on the sample virtual human image I pf With the corresponding sample human body image I pr The comparison results yield the first generator G. S The loss value is used to update the first generator G through back gradient calculation. S The weights. Based on the first probability P s The loss value is obtained by comparing it with the theoretical probability, and the first discriminator D is updated by calculating the backpropagation gradient. s The weights. Repeat the above steps until the first generator G. S And the first discriminator D s The loss value converges and the first discriminator D s Training is complete when the probability of the output is close to 50%.
[0110] In some embodiments, such as Figure 4 As shown, Figure 4This is a schematic diagram of the background fusion network training process provided in an embodiment of this application. The background fusion network includes a second generator and a second judge. The method includes:
[0111] Step S21: Obtain the sample target environment image, the sample real person working image, the sample human body coordinate information, and the sample virtual human body image, wherein the sample virtual human body image is generated by the human body generation network;
[0112] Step S22: Input the sample target environment image, the sample virtual human image, and the sample human coordinate information into the second generator to generate a sample virtual human working image;
[0113] Step S23: Input the sample virtual character working image or the sample real character working image into the second judge and output the second probability;
[0114] Step S24: Update the second generator by back gradient calculation based on the comparison results between the sample virtual character working image and the corresponding sample real character working image; update the weights of the second discriminator by back gradient calculation based on the second probability.
[0115] Step S25: Repeat the above steps until the background fusion network converges.
[0116] In the embodiments of this application, such as Figure 5 As shown, sample real-life work images I were obtained via the network. wr Sample target environment image I g Sample target environment image I g Sample virtual human body image I pf and sample human body coordinate information C xy As the second generator G c The input, the second generator G c Based on the human body coordinate information C xy Fusion Sample Target Environment Image I g Sample virtual human body image I pf Generate sample virtual character working image I wo The sample virtual character working image I wo Or sample real people working images I wr Input to the second discriminator D c Second discriminator D c Output the second probability P that belongs to the image of a real person at work. c Based on the sample virtual character working image I wo Corresponding real-life work image I wr The comparison results yield the second generator G. cThe loss value is used to update the second generator G through back gradient calculation. c The weights. Based on the second probability P. s The loss value is obtained by comparing it with the theoretical probability, and the second discriminator D is updated by calculating the backpropagation gradient. c The weights. Repeat the above steps until the second generator G. c Second discriminator D c The loss value converges and the second discriminator D c Training is complete when the probability of the output is close to 50%.
[0117] In some embodiments, before generating annotation information for the virtual character's working image and obtaining the virtual character's working image with annotation information, the method includes:
[0118] Step S7: Preprocess the virtual character working image, the preprocessing including random rotation, translation, brightness variation and / or noise addition.
[0119] In the embodiments of this application, image enhancement processing, such as random rotation, translation, brightness variation, and / or noise addition, is applied to the working image of the virtual character to facilitate later use.
[0120] In some embodiments, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the annotation generation process provided in the embodiments of this application. It describes how to generate annotation information for the virtual character working image to obtain the virtual character working image with annotation information, including:
[0121] Step S31: Obtain the human body coordinate information and behavior information of the virtual human body image in the virtual character working image, wherein the human body coordinate information includes the human body coordinate position of the virtual human body image in the target environment image and the width and height values of the virtual human body image, and the behavior information is the target behavior of the virtual human body image;
[0122] Step S32: Based on the human body coordinate information and behavior information, generate annotation information for the virtual character working image to obtain the virtual character working image with annotation information.
[0123] In the embodiments of this application, the annotation information refers to the human coordinate information and behavior information corresponding to the virtual human image in the virtual character's working image. The human coordinate information includes the human coordinate position of the virtual human image in the target environment image and the width and height of the human image. The behavior information includes the target behavior present in the virtual human image. The present target behavior can be a non-target behavior or a target behavior. The presence of target behavior can indicate which specific target behaviors are present, such as smoking, making a phone call, etc. The target behavior can be one or multiple, and the embodiments of this application do not limit this. When generating annotation information for the virtual character's working image, it is only necessary to extract the human coordinate information and behavior information of the virtual character's working image to generate the annotation information, without the need for manual annotation.
[0124] To address the problems existing in related technologies, this application provides a training method applied to an electronic device, such as a mobile phone or computer. The functionality achieved by the training method provided in this application can be implemented by the processor of the electronic device calling program code, which can be stored in a computer storage medium.
[0125] This application provides a training method, such as... Figure 7 As shown, Figure 7 This is a flowchart illustrating the training method provided in this application embodiment. Using the aforementioned sample training set, the detection model includes a human detection model and a behavior recognition model. The method includes:
[0126] Step S41: Statistically analyze the annotation information of all the virtual character working images in the sample training set, and obtain the width and height values of the labeled target boxes based on the annotation information;
[0127] Step S42: Perform K-Means clustering analysis on the width and height values of the target box to obtain the width and height values of multiple anchor boxes;
[0128] Step S43: Input the virtual character working images in the sample training set into the human body detection model, and obtain the predicted bounding boxes and confidence scores of the human body at different scales based on the width and height values of multiple anchor boxes;
[0129] Step S44: Calculate the predicted bounding box loss value and the confidence loss value, and perform gradient calculation to update the weights of the human detection model based on the predicted bounding box loss value and the confidence loss value;
[0130] Step S45: Repeat the above training steps until the predicted box loss value and confidence loss value converge or reach the specified number of training iterations;
[0131] Step S46: Input the working image of the virtual character exhibiting the target behavior into the human body detection model, and output the human body image;
[0132] Step S47: Input the human image into the behavior recognition model and output the third probability;
[0133] Step S48: Calculate the third loss value of the third probability, and update the weights of the behavior recognition model based on the gradient calculation of the third loss value;
[0134] Step S49: Repeat the above training steps until the third loss value converges or the specified number of training iterations are reached.
[0135] In the embodiments of this application, the detection model is a second-order detection model, consisting of a human detection model and a behavior recognition model. The human detection model uses the YOLOv5 object detection model, and the behavior recognition model is a multi-label classification model built using VisionTransformer. The annotation information of all virtual character working images in the training set is statistically analyzed. This annotation information includes the human coordinates and width and height of the virtual human image within the virtual character working image. The width and height values of the labeled bounding boxes are obtained based on the human coordinates and the width and height of the human image. K-Means clustering analysis is performed on the width and height values of the labeled bounding boxes of all virtual character working images in the training set, thereby classifying them. Each category corresponds to one anchor box, and the width and height values of the anchor box are obtained based on the width and height values of the labeled bounding boxes. The virtual character working images from the training dataset are input into the human detection model. Image features are extracted through convolution and pooling calculations to obtain prediction boxes and confidence scores for the human body at different scales. Different scales refer to image features at different magnifications, such as 2x, 4x, 8x, 16x, etc. The prediction box loss is obtained by comparing the predicted box with the corresponding target box, and the confidence loss is obtained by comparing the confidence score with a preset confidence level. The above training steps are repeated until the prediction box loss and confidence loss converge or reach a specified number of training iterations, at which point the human detection model is considered trained successfully. Virtual character working images labeled with target behavior from the sample training set are used as input to the trained human detection model, outputting human images. The behavior recognition model encodes the human images, extracts human features, and finally outputs a third probability that the human image contains the target behavior. A third loss value is calculated by comparing the third probability with a preset probability, and then gradient calculation is performed based on the third loss value to update the weights of the behavior recognition model. The above training steps are repeated until the third loss value converges or reaches a specified number of training iterations, at which point the behavior recognition model is considered trained successfully.
[0136] To address the problems existing in related technologies, this application provides a training method applied to an electronic device, such as a mobile phone or computer. The functionality achieved by the training method provided in this application can be implemented by the processor of the electronic device calling program code, which can be stored in a computer storage medium.
[0137] This application provides an identification method, such as... Figure 8 As shown, Figure 8 This is a flowchart illustrating the identification method provided in the embodiments of this application. Applying the detection model described above, the method includes:
[0138] Step S51: Acquire and decode the surveillance video of the target environment to obtain a single frame image;
[0139] Step S52: Perform adaptive image scaling and pixel normalization preprocessing on the single frame image;
[0140] Step S53: Input the preprocessed single-frame image into the detection model and output the probability of the existence of target behavior.
[0141] In the embodiments of this application, a port is taken as an example. A port is a hub and convergence point for land and water transportation, a distribution center for industrial and agricultural products and foreign trade import and export goods, and a place for ships to berth, load and unload cargo, embark and disembark passengers, and replenish supplies. Ports also have strict requirements for safety management, especially regarding the personal safety of staff. For example, smoking and making phone calls violate port safety management and can be considered as target behaviors. In implementation, a monitoring system is deployed in the port to collect port surveillance video. The video is decoded to obtain single-frame images, which are then preprocessed. The preprocessed single-frame images are input into a detection model. In the first stage, a human detection model is used to detect the location of human bodies in the preprocessed single-frame images and crop out the human body images. If a human body image is found, the process proceeds to the second stage. In the second stage, a behavior recognition model is used to identify the human body image and output the probability that a human body is smoking or making a phone call. If no human body image is found, the probability of either behavior is directly output as 0. In summary, applying the detection model to port operation risk monitoring can promptly detect whether relevant personnel are engaging in targeted behavior, reduce operational risks, and achieve high detection efficiency and accurate detection data.
[0142] In some embodiments, such as Figure 9 As shown, Figure 9 This is another flowchart illustrating the recognition method provided in this application embodiment. After inputting the preprocessed single-frame image into the detection model and outputting the probability of the existence of the target behavior, the method includes:
[0143] Step S54: Compare the probability with a preset threshold;
[0144] Step S55: If the probability is greater than the preset threshold, an alarm message is generated.
[0145] In the embodiments of this application, if the probability of detecting target behavior by relevant personnel is greater than a corresponding threshold, an operational risk is considered to exist, and a corresponding alarm is triggered; if the probability is less than the corresponding threshold, no alarm is triggered. This achieves proactive early warning and reduces operational risks.
[0146] This application provides a sample generation device, such as... Figure 10 As shown, Figure 10 A schematic diagram of the sample generation apparatus provided in the embodiments of this application includes:
[0147] The first acquisition module is used to acquire the first person image with target behavior, the second person image without target behavior, and the target environment image;
[0148] The extraction module is used to extract the pose information of the first person image and the second person image;
[0149] The first generation module is used to input the posture information and normally distributed random noise into the human body generation network to generate a virtual human body image that conforms to the posture information.
[0150] The second generation module is used to input the virtual human image, the target environment image and human coordinate information into the background fusion network to generate a virtual human working image.
[0151] The annotation module is used to generate annotation information for the virtual character's working image, thereby obtaining the virtual character's working image with annotation information;
[0152] The third generation module is used to generate a sample training set based on the virtual character's working image with labeled information.
[0153] In some embodiments, the human body generation network includes a first generator and a first judge, and the sample generation device includes:
[0154] The third acquisition module is used to acquire a first sample human image that exhibits the target behavior and a second sample human image that does not exhibit the target behavior.
[0155] The second extraction module is used to extract the real human body image of the first sample person image, the real human body image of the second sample person image, and the human body posture information of the sample person image.
[0156] The fourth generation module is used to input the sample human posture information and sample random noise that conforms to a normal distribution into the first generator to generate a sample virtual human image that conforms to the sample posture information.
[0157] The fifth input module is used to input the real human body image or the virtual human body image of the sample into the first discriminator and output the first probability;
[0158] The first update module is used to update the weights of the first generator by back gradient calculation based on the comparison results between the sample virtual human image and the corresponding first sample human image or second sample human image, and to update the weights of the first discriminator by back gradient calculation based on the first probability.
[0159] The third repetition module is used to repeat the above steps until the human body generation network converges.
[0160] In some embodiments, the background fusion network includes a second generator and a second determiner, and the sample generation device includes:
[0161] The fourth acquisition module is used to acquire sample target environment image, sample real person working image, sample human body coordinate information and sample virtual human body image, wherein the sample virtual human body image is generated by the human body generation network;
[0162] The fifth generation module is used to input the sample target environment image, the sample virtual human image, and the sample human coordinate information into the second generator to generate a sample virtual human working image;
[0163] The sixth input module is used to input the sample virtual character working image or the sample real character working image into the second judge and output the second probability;
[0164] The second update module is used to update the second generator by back gradient calculation based on the comparison results between the sample virtual character working image and the corresponding sample real character working image, and to update the weight of the second discriminator by back gradient calculation based on the second probability.
[0165] The fourth repeating module is used to repeat the above steps until the background fusion network converges.
[0166] In some embodiments, before generating annotation information for the virtual character working image and obtaining the virtual character working image with annotation information, the sample generation device includes:
[0167] The first preprocessing module is used to preprocess the working image of the virtual character. The preprocessing includes random rotation, translation, brightness variation and / or noise addition.
[0168] In some embodiments, annotation information is generated for the virtual character working image to obtain the virtual character working image with annotation information, and the sample generation device includes:
[0169] The acquisition unit is used to acquire the human coordinate information and behavior information of the virtual human image in the virtual character working image, wherein the human coordinate information includes the human coordinate position of the virtual human image in the target environment image and the width and height values of the virtual human image, and the behavior information is the target behavior of the virtual human image;
[0170] The annotation unit is used to generate annotation information for the virtual character working image based on the human body coordinate information and behavior information, so as to obtain the virtual character working image with annotation information.
[0171] This application provides a training device, such as... Figure 11 As shown, Figure 11 This is a schematic diagram of the training device provided in the embodiments of this application. Using the aforementioned sample training set, the detection model includes a human detection model and a behavior recognition model, comprising:
[0172] The statistics module is used to collect the annotation information of all the virtual character working images in the sample training set, and obtain the width and height values of the target box for annotation based on the annotation information;
[0173] The clustering analysis module is used to perform K-Means clustering analysis on the width and height values of the target boxes to obtain the width and height values of multiple anchor boxes;
[0174] The first input module is used to input the virtual character working images in the sample training set into the human body detection model, and obtain the prediction boxes and confidence scores of the human body at different scales based on the width and height values of multiple anchor boxes.
[0175] The first calculation module is used to calculate the prediction box loss value and the confidence loss value, and to perform gradient calculation to update the weights of the human detection model based on the prediction box loss value and the confidence loss value.
[0176] The first repetition module is used to repeat the above training steps until the prediction box loss value and the confidence loss value converge or reach the specified number of training times.
[0177] The second input module is used to input the working image of the virtual character with the target behavior into the human body detection model and output a human body image.
[0178] The third input module is used to input the human image into the behavior recognition model and output a third probability that the target behavior exists;
[0179] The second calculation module is used to calculate the third loss value based on the third probability, and to perform gradient calculation to update the weights of the behavior recognition model based on the third loss value.
[0180] The second repetition module is used to repeat the above training steps until the third loss value converges or the specified number of training iterations are reached.
[0181] This application provides an identification device, such as... Figure 12 As shown, Figure 12 This is a schematic diagram of the structure of the identification device provided in the embodiments of this application. Applying the detection model described above, it includes:
[0182] The second acquisition module is used to acquire and decode the surveillance video of the target environment to obtain a single frame image;
[0183] The preprocessing module is used to perform adaptive image scaling and pixel normalization preprocessing on single-frame images;
[0184] The fourth input module is used to input the preprocessed single-frame image into the detection model and output the probability of the existence of target behavior.
[0185] In some embodiments, after inputting the preprocessed single-frame image into the detection model and outputting the probability of the presence of target behavior, the recognition device:
[0186] The comparison module is used to compare the probability with a preset threshold.
[0187] An alarm module is used to generate an alarm message if the probability is greater than the preset threshold.
[0188] It should be noted that, in the embodiments of this application, if the above-mentioned method for finding virtual items is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.
[0189] Accordingly, this application provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the sample generation method, training method, and recognition method provided in the above embodiments.
[0190] This application provides an electronic device; Figure 13 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application, such as... Figure 13 As shown, the electronic device 100 includes: a processor 101, at least one communication bus 102, a user interface 103, at least one external communication interface 104, and a memory 105. The communication bus 102 is configured to enable communication between these components. The user interface 103 may include a display screen, and the external communication interface 104 may include standard wired and wireless interfaces. The processor 101 is configured to execute programs stored in the memory for sample generation, training, and recognition methods to implement the steps in the sample generation, training, and recognition methods provided in the above embodiments.
[0191] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.
[0192] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.
[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0194] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0195] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0196] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0197] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.
[0198] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a controller to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0199] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A sample generation method, characterized in that, The method includes: Acquire the image of the first person exhibiting the target behavior, the image of the second person not exhibiting the target behavior, and the image of the target environment; Extract the pose information of the first person image and the second person image; The posture information and normally distributed random noise are input into the human body generation network to generate a virtual human body image that conforms to the posture information. The virtual human image, the target environment image, and human coordinate information are input into a background fusion network to generate a virtual character working image. The background fusion network includes a second generator and a second discriminator. Generate annotation information for the virtual character working image to obtain the virtual character working image with annotation information, which includes: obtaining human coordinate information and behavior information of the virtual human image in the virtual character working image, wherein the human coordinate information includes the human coordinate position of the virtual human image in the target environment image and the width and height values of the virtual human image, and the behavior information is the target behavior existing in the virtual human image; and generating annotation information for the virtual character working image based on the human coordinate information and behavior information to obtain the virtual character working image with annotation information. A sample training set is generated based on the virtual character's working images with labeled information; The human body generation network includes a first generator and a first discriminator. The method includes: acquiring a first sample human image with a target behavior and a second sample human image without a target behavior; extracting real human images and human pose information from the first and second sample human images; inputting the human pose information and normally distributed random noise into the first generator to generate a virtual human image that conforms to the human pose information; inputting the real human image or the virtual human image into the first discriminator and outputting a first probability; updating the weights of the first generator by backpropagation gradient calculation based on the comparison result between the virtual human image and the corresponding first or second sample human image, and updating the weights of the first discriminator by backpropagation gradient calculation based on the first probability; repeating the above steps until the human body generation network converges.
2. The method according to claim 1, characterized in that, The method includes: The sample target environment image, sample real person working image, sample human body coordinate information, and sample virtual human body image are acquired, wherein the sample virtual human body image is generated by the human body generation network; The sample target environment image, the sample virtual human image, and the sample human coordinate information are input into the second generator to generate a sample virtual human working image. The sample virtual character working image or the sample real character working image is input into the second discriminator, and the second probability is output; The second generator is updated by back gradient calculation based on the comparison results between the sample virtual character working image and the corresponding sample real character working image, and the weights of the second discriminator are updated by back gradient calculation based on the second probability. Repeat the above steps until the background fusion network converges.
3. The method according to claim 1, characterized in that, Before generating annotation information for the virtual character's working image and obtaining the virtual character's working image with annotation information, the method includes: The virtual character working image is preprocessed, including random rotation, translation, brightness variation and / or noise addition.
4. A method for training a detection model, characterized in that, The sample training set generated using the sample generation method according to any one of claims 1-3, the detection model includes a human detection model and a behavior recognition model, and the method includes: The annotation information of all the virtual character working images in the sample training set is statistically analyzed, and the width and height values of the labeled target boxes are obtained based on the annotation information; K-Means clustering analysis was performed on the width and height values of the target box to obtain the width and height values of multiple anchor boxes; The virtual character working images in the sample training set are input into the human detection model. Based on the width and height values of multiple anchor boxes, the predicted bounding boxes and confidence scores of the human body at different scales are obtained. Calculate the predicted bounding box loss value and the confidence loss value, and perform gradient calculation to update the weights of the human detection model based on the predicted bounding box loss value and the confidence loss value; Repeat the above training steps until the predicted box loss value and the confidence loss value converge or the specified number of training iterations are reached. The working image of the virtual character exhibiting the target behavior is input into the human body detection model, and a human body image is output. The human image is input into the behavior recognition model, and a third probability is output. Calculate the third loss value of the third probability, and update the weights of the behavior recognition model based on the gradient calculation of the third loss value; Repeat the above training steps until the third loss value converges or the specified number of training iterations are reached.
5. A method for identification, characterized in that, The detection model is trained using the training method of the detection model according to claim 4, wherein the method includes: Acquire and decode surveillance video of the target environment to obtain single-frame images; Preprocessing of single-frame images includes adaptive image scaling and pixel normalization; The preprocessed single-frame image is input into the detection model, which outputs the probability of the presence of the target behavior.
6. The method according to claim 5, characterized in that, After inputting the preprocessed single-frame image into the detection model and outputting the probability of the presence of target behavior, the method includes: Compare the probability with a preset threshold; If the probability is greater than the preset threshold, an alarm message is generated.
7. A sample generation device, characterized in that, include: The first acquisition module is used to acquire the first person image with target behavior, the second person image without target behavior, and the target environment image; The extraction module is used to extract the pose information of the first person image and the second person image; The first generation module is used to input the posture information and normally distributed random noise into the human body generation network to generate a virtual human body image that conforms to the posture information. The second generation module is used to input the virtual human image, the target environment image and human coordinate information into the background fusion network to generate a virtual human working image. The background fusion network includes a second generator and a second discriminator. The annotation module is used to generate annotation information for the virtual character's working image, thereby obtaining the virtual character's working image with annotation information; The annotation module is used to obtain human coordinate information and behavior information of the virtual human image in the virtual character working image. The human coordinate information includes the human coordinate position of the virtual human image in the target environment image and the width and height values of the virtual human image. The behavior information is the target behavior of the virtual human image. Based on the human coordinate information and behavior information, annotation information is generated for the virtual character working image to obtain the virtual character working image with annotation information. The third generation module is used to generate a sample training set based on the virtual character's working image with labeled information; The human body generation network includes a first generator and a first discriminator. The device is further configured to: acquire a first sample human image with target behavior and a second sample human image without target behavior; extract sample real human images and sample human pose information from the first sample human image and the second sample human image; input the sample human pose information and sample random noise conforming to a normal distribution into the first generator to generate a sample virtual human image conforming to the sample human pose information; input the sample real human image or the sample virtual human image into the first discriminator and output a first probability; update the weights of the first generator by backpropagation gradient calculation based on the comparison result between the sample virtual human image and the corresponding first sample human image or second sample human image, and update the weights of the first discriminator by backpropagation gradient calculation based on the first probability; repeat the above steps until the human body generation network converges.
8. A training device for a detection model, characterized in that, The sample training set generated using the sample generation method according to any one of claims 1-3, the detection model includes a human detection model and a behavior recognition model, comprising: The statistics module is used to collect the annotation information of all the virtual character working images in the sample training set, and obtain the width and height values of the target box for annotation based on the annotation information; The clustering analysis module is used to perform K-Means clustering analysis on the width and height values of the target boxes to obtain the width and height values of multiple anchor boxes; The first input module is used to input the virtual character working images in the sample training set into the human body detection model, and obtain the prediction boxes and confidence scores of the human body at different scales based on the width and height values of multiple anchor boxes. The first calculation module is used to calculate the prediction box loss value and the confidence loss value, and to perform gradient calculation to update the weights of the human detection model based on the prediction box loss value and the confidence loss value. The first repetition module is used to repeat the above training steps until the prediction box loss value and the confidence loss value converge or reach the specified number of training times. The second input module is used to input the working image of the virtual character with the target behavior into the human body detection model and output a human body image. The third input module is used to input the human image into the behavior recognition model and output a third probability that the target behavior exists; The second calculation module is used to calculate the third loss value based on the third probability, and to perform gradient calculation to update the weights of the behavior recognition model based on the third loss value. The second repetition module is used to repeat the above training steps until the third loss value converges or the specified number of training iterations are reached.
9. An identification device, characterized in that, The detection model trained using the training method of the detection model according to claim 4 includes: The second acquisition module is used to acquire and decode the surveillance video of the target environment to obtain a single frame image; The preprocessing module is used to perform adaptive image scaling and pixel normalization preprocessing on single-frame images; The fourth input module is used to input the preprocessed single-frame image into the detection model and output the probability of the existence of target behavior.
10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program that, when executed by the processor, performs the method as described in any one of claims 1 to 6.
11. A storage medium, characterized in that, The computer program stored in the storage medium can be executed by one or more processors and can be used to implement the method as described in any one of claims 1 to 6.