Image processing model training method, device and electronic equipment

Abstract

A method, apparatus, and electronic device for training an image processing model, the method comprising: performing multiple iterative training on a first image processing model, and determining a target image processing model corresponding to the first image processing model based on the image processing models obtained during each of the multiple iterative trainings and the loss percentage corresponding to each of the multiple iterative trainings. Through the above-described method, after obtaining the first image processing model, the first image processing model is subjected to multiple iterative training. The image processing model with the most accurate prediction results is screened out from the multiple iterative image processing models and is used as the target image processing model, thereby ensuring the accuracy of the target image processing model.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a training method, device and electronic equipment for an image processing model. Background Art

[0002] To ensure the safety of workers at power maintenance sites and prevent falls from heights, workers are required to wear safety belts. To improve the efficiency of safety belt monitoring, an intelligent safety belt detection system is required.

[0003] At present, in order to detect whether workers are wearing safety belts, one method used is to use a deep convolutional neural network to distinguish between helmets and safety belts. The deep convolutional neural network model and the spatial correlation model used to lock the helmets and safety belts are used to detect the worker images in the video and distinguish whether they are wearing helmets and safety belts. The image processing model is obtained based on the deep convolutional neural network model and the spatial correlation model. However, during the training of the deep convolutional neural network, the image processing model extracts image features again by reducing and increasing the number of channels. When the number of channels increases, more image features will be obtained, which will lead to overfitting of the image processing model during the training process. Although overfitting will make the prediction results of the image processing model more accurate during the training process, when the image processing model is used for actual prediction, the actual prediction result accuracy is reduced. When the image features are used to detect whether the worker is wearing a safety belt, the worker cannot be accurately locked, resulting in low detection accuracy of the image processing model. Summary of the Invention

[0004] The present application provides a training method, device and electronic device for an image processing model, in which the i+1th image processing model is iteratively trained using the i-th loss ratio. After the i+1th image processing model is trained, the i+1th loss ratio is obtained. The image processing model with the highest prediction accuracy is screened out from multiple image processing models that are iteratively trained, and the image processing model is used as the target image processing model, thereby improving the accuracy of the detection results of the target image processing model.

[0005] In a first aspect, the present application provides a method for training an image processing model, the method comprising:

[0006] Performing multiple iterative training on a first image processing model, and determining a target image processing model corresponding to the first image processing model based on an image processing model obtained by each iterative training in the multiple iterative training and a loss ratio corresponding to each iterative training, wherein the i+1th iterative training in the multiple iterative training includes:

[0007] Performing sample expansion processing on at least part of the i-th sample image set used in the i+1-th iterative training according to the i-th loss ratio corresponding to the i-th iterative training to obtain at least one expanded sample;

[0008] Based on the at least one extended sample and the i-th sample image set, the i-th image processing model obtained by the i-th iterative training is trained to obtain the i+1-th image processing model corresponding to the i+1-th iterative training and the i+1-th loss ratio corresponding to the i+1-th iterative training, where i is a positive integer.

[0009] In one possible design, determining a target image processing model corresponding to the first image processing model based on an image processing model obtained from each iterative training in multiple iterative trainings and a loss ratio corresponding to each iterative training includes:

[0010] Obtain M image processing models trained over multiple iterations and M loss ratios, where M is a positive integer;

[0011] The image processing model corresponding to the minimum loss ratio is used as the target image processing model.

[0012] In one possible design, the process of obtaining the first image processing model includes:

[0013] The first sample image is input into a preset network for training to obtain a first image processing model.

[0014] In one possible design, based on the i-th loss ratio corresponding to the i-th iterative training, performing sample expansion processing on at least part of the i-th sample images in the i-th sample image set used in the i+1-th iterative training to obtain at least one expanded sample includes:

[0015] Determining whether the received i-th loss ratio exceeds a preset threshold;

[0016] If so, scaling at least part of the i-th sample images in the i-th sample image set to obtain a plurality of scaled images, and splicing scaled images with the same resolution among the plurality of scaled images to obtain at least one first extended sample, and determining the first extended sample as the i-th sample image in the i-th sample image set;

[0017] If not, rotate at least part of the i-th sample image in the i-th sample image set to obtain multiple rotated images, and add the pixels of the rotated images of the same size in the multiple rotated images according to the image weight value to obtain at least one second extended sample, and determine the second extended sample as the i-th sample image in the i-th sample image set.

[0018] In a possible design, before training the i-th image processing model obtained by the i-th iterative training, the following steps are performed:

[0019] Converting the at least one extended sample in the RGB space and each image in the i-th sample image set into the HIS space;

[0020] Decomposing each image into a high-frequency component and a low-frequency component according to brightness, the high-frequency component is processed by linear weighted enhancement, and the low-frequency component is processed by histogram equalization;

[0021] The high-frequency components processed by linear weighted enhancement and the low-frequency components processed by histogram equalization in each image are fused, and each fused image is inversely transformed from the HIS space to the RGB space.

[0022] In one possible design, based on the at least one extended sample and the i-th sample image set, the i-th image processing model obtained by the i-th iterative training is trained to obtain the i+1-th image processing model corresponding to the i+1-th iterative training and the i+1-th loss ratio corresponding to the i+1-th iterative training, including:

[0023] Inputting the at least one extended sample and the i-th sample image set into a prediction network for training to obtain an image feature set of the i+1-th sample image set, wherein the images in the image feature set have different resolutions;

[0024] Prediction training is performed on the i+1th sample image set based on the image feature set to obtain the i+1th loss ratio.

[0025] In one possible design, performing prediction training on the (i+1)th sample image set based on the image feature set includes:

[0026] Acquire a target object image in the (i+1)th sample image set, and detect whether the image area of ​​the target object image exceeds a preset area;

[0027] If so, the target object image exceeding the preset area is used as the first target object image;

[0028] If not, the target object image smaller than the preset area is used as the second target object image;

[0029] Prediction training is performed on the first target object image and the second target object image based on the feature image set according to preset rules.

[0030] In one possible design, performing prediction training on the first target object image and the second target object image based on the feature image according to a preset rule includes:

[0031] Extracting a first positive sample and a first negative sample corresponding to the first target object image and a second positive sample and a second negative sample corresponding to the second target object image, respectively, wherein the positive sample is an image corresponding to a specified area in the target object image, and the negative sample is an image corresponding to an area outside the specified area in the target object image;

[0032] When the target object image is a first target object image, predicting the first positive sample and the first negative sample using a feature image of a first resolution;

[0033] When the target object image is a second target object image, the second positive sample and the second negative sample are predicted using a feature image of a second resolution.

[0034] In a second aspect, the present application provides a method for behavior recognition, the method comprising:

[0035] Based on the trained target image processing model, behavior recognition is performed on the image to be processed containing the target object to determine whether the target object has abnormal behavior, wherein the abnormal behavior is that the target object is not wearing a seat belt. The target image processing model is obtained by training the first image processing model.

[0036] In one possible design, behavior recognition is performed on an image to be processed containing a target object based on a trained target image processing model to determine whether the target object has abnormal behavior, including:

[0037] If it is determined that the target object in the current frame has abnormal behavior, an alarm message is output;

[0038] If it is determined that the identification information of the target object in the next frame of the current frame is inconsistent with the identification information of the target object in the current frame, an alarm message is output.

[0039] In a third aspect, the present application provides a training device for an image processing model, the device comprising:

[0040] an iterative module, configured to perform multiple iterative training on the first image processing model, and determine a target image processing model corresponding to the first image processing model based on the image processing model obtained from each iterative training and the loss ratio corresponding to each iterative training in the multiple iterative training;

[0041] an expansion module, configured to perform sample expansion processing on at least part of the i-th sample image set used in the i+1-th iterative training according to the i-th loss ratio corresponding to the i-th iterative training, to obtain at least one expanded sample;

[0042] A training module is used to train the i-th image processing model obtained by the i-th iterative training based on the at least one extended sample and the first sample image set, to obtain the i+1-th image processing model corresponding to the i+1-th iterative training and the i+1-th loss ratio corresponding to the i+1-th iterative training.

[0043] In one possible design, the iteration module is specifically used to obtain M image processing models and M loss ratios trained through multiple iterations, and to take the image processing model corresponding to the minimum loss ratio as the target image processing model.

[0044] In a possible design, the iteration module is further used to input the first sample image into a preset network for training to obtain a first image processing model.

[0045] In one possible design, the expansion module is specifically used to determine whether the received i-th loss ratio exceeds a preset threshold. If so, at least part of the i-th sample image in the i-th sample image set is scaled to obtain multiple scaled images, and scaled images of the same resolution in the multiple scaled images are spliced ​​to obtain at least one first extended sample, and the first extended sample is determined as the i-th sample image in the i-th sample image set. If not, at least part of the i-th sample image in the i-th sample image set is rotated to obtain multiple rotated images, and pixels of the rotated images of the same size in the multiple rotated images are added according to the weight value of the image to obtain at least one second extended sample, and the second extended sample is determined as the i-th sample image in the i-th sample image set.

[0046] In one possible design, the expansion module is further used to convert the at least one extended sample in the RGB space and each image in the i-th sample image set into the HIS space, decompose each image into a high-frequency component and a low-frequency component according to brightness, process the high-frequency component with linear weighted enhancement, and process the low-frequency component with histogram equalization, fuse the high-frequency component processed with linear weighted enhancement and the low-frequency component processed with histogram equalization in each image, and invert each fused image from the HIS space to the RGB space.

[0047] In one possible design, the training module is specifically used to input the at least one extended sample and the i-th sample image set into the prediction network for training, obtain the image feature set of the i+1-th sample image set, perform prediction training on the i+1-th sample image set based on the image feature set, and obtain the i+1-th loss ratio.

[0048] In one possible design, the training module is also used to obtain the target object image in the i+1th sample image set, detect whether the image area of ​​the target object image exceeds a preset area, and if so, use the target object image exceeding the preset area as the first target object image; if not, use the target object image below the preset area as the second target object image, and perform prediction training on the first target object image and the second target object image based on the feature image set according to preset rules.

[0049] In one possible design, the training module is also used to extract a first positive sample and a first negative sample corresponding to the first target object image and a second positive sample and a second negative sample corresponding to the second target object image, respectively, wherein the positive sample is an image corresponding to a specified area in the target object image, and the negative sample is an image corresponding to an area outside the specified area in the target object image. When the target object image is the first target object image, the first positive sample and the first negative sample are predicted using a feature image of a first resolution; when the target object image is the second target object image, the second positive sample and the second negative sample are predicted using a feature image of a second resolution.

[0050] In a fourth aspect, the present application provides a device for behavior recognition, the device comprising:

[0051] The recognition module is used to perform behavior recognition on the image to be processed containing the target object based on the trained target image processing model to determine whether the target object has abnormal behavior.

[0052] In one possible design, the recognition module is specifically used to output an alarm message if it determines that the target object in the current frame has abnormal behavior, and output an alarm message if it determines that the identification information of the target object in the next frame of the current frame is inconsistent with the identification information of the target object in the current frame.

[0053] In a fifth aspect, the present application provides an electronic device, comprising:

[0054] Memory for storing computer programs;

[0055] The processor is used to implement the above-mentioned image processing model training method steps and behavior recognition method steps when executing the computer program stored in the memory.

[0056] In a sixth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned training method steps of an image processing model and the method steps of a behavior recognition.

[0057] For each of the above-mentioned aspects 1 to 6 and the technical effects that may be achieved by each of the aspects, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or various possible solutions in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart of the steps of a training method for an image processing model provided in this application;

[0059] Figure 2 A flowchart for performing convolution on the second sample image set provided by this application;

[0060] Figure 3 A schematic diagram of the acquisition process of the feature image set corresponding to the second sample image set provided in this application;

[0061] Figure 4 A schematic diagram of the mapping process of the second sample image provided in this application;

[0062] Figure 5 A schematic diagram of the prediction network output prediction results provided by this application;

[0063] Figure 6 A diagram showing the training process of the prediction network provided in this application during the image processing model training process;

[0064] Figure 7 Schematic diagram of the positive sample provided for this application;

[0065] Figure 8 A flowchart of the steps of a behavior recognition method provided in this application;

[0066] Figure 9 A schematic diagram of the structure of a training device for an image processing model provided in this application;

[0067] Figure 10 A schematic diagram of the structure of a behavior recognition device provided in this application;

[0068] Figure 11 This is a schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail with reference to the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent the following two situations: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0070] In previous technologies, in order to detect whether workers are wearing seat belts, a detection method for identifying safety helmets and seat belts based on deep convolutional neural networks was adopted. The images of workers in the video were detected and the wearing status of their safety helmets and seat belts was identified through a deep convolutional neural network model and a spatial correlation model for locking safety helmets and seat belts. An image processing model was obtained based on the deep convolutional neural network model and the spatial correlation model. However, during the training of the deep convolutional neural network, the image processing model extracted image features again by reducing and increasing the number of channels. When the number of channels increased, more image features would be obtained, which would lead to overfitting of the image processing model during the training process. Therefore, when the image features were used to detect whether workers were wearing seat belts, the workers could not be accurately locked, resulting in low accuracy in the detection results of the image processing model.

[0071] To address the aforementioned issues, the present invention provides a method for training an image processing model to address the low accuracy of the image processing model's detection results, thereby improving the accuracy of the image processing model in detecting whether a worker is wearing a seat belt. The method and device described in the present invention are based on the same technical concept. Since the principles of the problems solved by the method and device are similar, the embodiments of the device and method can refer to each other, and any repetitions will not be repeated.

[0072] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0073] Example 1

[0074] Reference Figure 1 This application provides a seat belt detection method that can improve the accuracy of the detection results of the image processing model. The implementation process of the method is as follows:

[0075] Step S1: Perform multiple iterative training on the first image processing model, and determine the target image processing model corresponding to the first image processing model based on the image processing model obtained in each iterative training and the loss ratio corresponding to each iterative training.

[0076] In an embodiment of the present application, it is necessary to improve the accuracy and detection speed of the image processing model in detecting whether a worker is wearing a seat belt. This requires increasing the number of images in the first sample image set so that the image processing model can obtain more feature images during training. Because the convolutional layer in the image processing model consumes a lot of computer resources, in order to increase the speed at which the computer reads the image data of the convolutional layer, improve the accuracy of the image processing model's detection results when detecting whether a worker is wearing a seat belt, and increase the detection speed of the image processing model, the embodiment of the present application uses 6 convolutional layers and sets the number of input channels and output channels of the convolutional layer to be consistent, thereby increasing the speed at which the computer reads the image data of the convolutional layer.

[0077] It should be noted here that the functions of the convolutional layer of the image processing model include: reducing the resolution of the image, classifying the image, etc.

[0078] The target image processing model is the image processing model with the smallest loss ratio selected after the image processing model has been trained M times. The loss ratio must be obtained after one model training. When the image processing model is trained for the first time, the number of images in the first sample image set is not increased.

[0079] Before describing the second model training process, it is necessary to obtain a second sample image set for model training. The second sample image set is obtained by adding at least one first extended sample or at least one second extended sample to the first sample image set. Therefore, the first sample image set must be obtained first. The process of obtaining the first sample image set is as follows:

[0080] The video data of the staff is obtained. The video data is composed of each frame of image. In order to avoid intercepting the images of adjacent frames, which causes a high degree of duplication between the images of adjacent frames, the images will be intercepted according to the preset period. After intercepting the images according to the preset period, the intercepted images will be used as the initial sample image set.

[0081] After obtaining the initial sample image set, in order to reduce the interference of background images in the initial sample image set, it is necessary to determine the target object image from each image in the initial sample image set. The target object image is a human body image, a seat belt image, etc. Therefore, it is necessary to cut out the target object image from each image in the initial sample image set to obtain the target object image set, delete the blank and blurred images in the target object image set, and use the remaining target object images as the first sample image set.

[0082] After the first image processing model training, the first loss ratio of the first sample image set will be obtained. The loss ratio indicates the accuracy of the image processing model. The smaller the loss ratio, the more accurate the image processing model will be in detecting whether the staff is wearing a seat belt.

[0083] It should be further explained that the first loss ratio obtained after the first image processing model training is the difference between the true value and the predicted value. The larger the predicted value, the smaller the first loss ratio.

[0084] The predicted value here reflects the accuracy of the image processing model training in predicting whether the staff is wearing a seat belt, and the true value represents the accuracy of detecting whether the staff is wearing a seat belt calculated after labeling the first sample image set.

[0085] In the embodiment of the present application, the specific process of labeling the first sample image set and calculating the true value of the first sample image set based on the labeling information is as follows:

[0086] Obtain the coordinates of the intersection of the left edge and the upper edge of the first sample image set, as well as the coordinates of the intersection of the right edge and the lower edge of the first sample image set, and record the category names of the target object images in all the first sample images in the first sample image set. The category names are represented by numbers. The relationship between the category names and numbers is shown in Table 1:

[0087] Category Name number people 0 seat belt 1 ...... ......

[0088] Table 1

[0089] According to Table 1, the numbers corresponding to the type names of the target object images can be obtained. The category names in Table 1 only list people and seat belts. The number corresponding to people is 0, and the number corresponding to seat belts is 1. For other category names and the numbers corresponding to the category names, please refer to the people and seat belts in Table 1.

[0090] According to the above description, all the first sample images in the first sample image set are labeled, and the two coordinates corresponding to each first sample image in the first sample image set and the numbers corresponding to the type name of the target object image in the first sample image are recorded. For example, the coordinates of image 1 are (0, 0) and (2, 2), and the number corresponding to the image category name is 0, so image 1 can be represented by (0, 0), (2, 2), and 0.

[0091] After labeling each first sample image in the first sample image set, in order to calculate the true value of the first sample image set, it is necessary to record the first sample images in the first sample image set and the labeling information corresponding to the first sample images. The labeling information corresponding to the first sample image set and each first sample image is shown in Table 2:

[0092] The first sample image set Annotation information corresponding to the first sample image Image 1 (0，0)、(2，2)、1 Image 2 (0，1)、(3，2)、0 Image 3 (1，1)、(3，3)、1 Image 4 (1，2)、(4，3)、3 ...... ......

[0093] Table 2

[0094] It should be noted in Table 2 that the first sample images in the first sample image set are associated with the annotation information corresponding to the first sample images. Table 2 only lists four first sample images and the annotations corresponding to the four first sample images. The other first sample images can refer to the form of any one of the four images.

[0095] After obtaining the annotation information of each first sample image in the first sample image set, prediction training is performed on the target object image based on the first sample image set, and then the true value corresponding to the first sample image set is calculated according to the annotation information. After obtaining the true value of the first sample image set, the first loss ratio of the first sample image set is calculated according to the true value and the predicted value.

[0096] Through the above method, the first sample image set is trained to obtain the first image processing model corresponding to the first sample image set, the first image processing model is trained multiple times through iterations, and the target image processing model with the highest prediction accuracy is screened out from the image processing models of the multiple iterations, thereby ensuring the accuracy of the target image processing model.

[0097] Step S2: performing sample expansion processing on at least part of the i-th sample images in the i-th sample image set used in the i+1-th iterative training according to the i-th loss ratio corresponding to the i-th iterative training, to obtain at least one extended sample.

[0098] In Example 1 of the present application, except that the first sample image in the first sample image set is not expanded during the training process of the first image processing model, the iterative training process of other image processing models is consistent. The iterative training process of the image processing model is to perform sample expansion processing on at least part of the i-th sample image in the i-th sample image set used in the i+1-th iterative training based on the i-th loss ratio corresponding to the i-th iterative training to obtain at least one extended sample. Taking the training process of the second image processing model as an example, the image processing model training process in other iterative processes refers to the training process of the second image processing model. The specific process of the second image processing model training is as follows:

[0099] After obtaining the first loss ratio corresponding to the first sample image set, during the second model training process, it is necessary to first detect whether the first loss ratio exceeds the preset threshold. If the first loss ratio exceeds the preset threshold, it is necessary to scale at least part of the first sample images in the first sample image set to obtain multiple scaled images, and splice the scaled images of the same resolution in the multiple scaled images to obtain at least one first extended sample, and use the at least one extended sample as the first sample image in the first sample image set. The stitching process is to stitch 4 images with the same resolution. After the stitching process is completed, the size of the image will become four times the original size.

[0100] If the first loss ratio is lower than a preset threshold, the first sample image is rotated to obtain multiple rotated images, and the pixels of the rotated images of the same size in the multiple rotated images are added according to the weight value of the image to obtain at least one second extended sample, and the at least one second extended sample is determined as the first sample image in the first sample image set. After adding the pixels of images of the same size according to the weight value of the image, the resolution of the image after the pixels are added will be higher. The resolution represents the density of pixels per unit area, and the size of the image does not change. Since adding pixels of multiple images according to the weight value of the image is a technology well known to those skilled in the art, it will not be elaborated here.

[0101] After all images of at least one first extended sample or at least one second extended sample are added to the first sample image set, in order to make the details of all first sample images in the first sample image set clearer, it is necessary to convert all first sample images from the RGB space to the HIS space. The RGB space is a color space, and the HIS space is a space that describes color from a human visual perspective using hue, color saturation, and brightness.

[0102] After all first sample images are converted from RGB space to HIS space, each first sample image is decomposed according to the histogram equalization function, and each first sample image is decomposed into a low-frequency component and a high-frequency component according to brightness. The low-frequency component represents the area in the image where the brightness changes slowly, and the high-frequency component represents the area in the image where the brightness changes dramatically. The grayscale change area corresponding to the low-frequency component is adjusted to make the grayscale distribution corresponding to the low-frequency component average. The high-frequency component is linearly weighted enhanced to increase the contrast of the image and make the details of the image clearer. After obtaining the high-frequency component and the low-frequency component of the first sample image, the high-frequency component and the low-frequency component need to be fused. After fusion, the first sample image is inversely transformed from the HIS space to the RGB space.

[0103] After the first sample image set is processed as above, the processed image set is used as the second sample image set.

[0104] Through the method described above, the number of first sample images in the first sample image set is increased to obtain at least one extended sample, and the at least one extended sample is used as the second sample image in the second sample image set, ensuring that the number of images in the second sample image set is sufficient. In the process of image feature extraction based on the second sample image set, it is beneficial to obtain more image features.

[0105] Step S2: Based on the at least one extended sample and the i-th sample image set, the i-th image processing model obtained by the i-th iterative training is trained to obtain the i+1-th image processing model corresponding to the i+1-th iterative training and the i+1-th loss ratio corresponding to the i+1-th iterative training.

[0106] After obtaining the second sample image set, in order to improve the accuracy of the image processing model in detecting whether the staff are wearing seat belts, the second sample image set needs to be input into the prediction network for training. The specific process of inputting the second sample image set into the prediction network for prediction training is as follows:

[0107] The second sample image in the second sample image set is input into the convolution layer, such as Figure 2 As shown, Figure 2 This is a flow chart for convolution of the second sample image. After each convolution, the resolution of the second sample image will decrease. Figure 2 After the first convolution, the image resolution is 80*80, after the second convolution, the image resolution is 40*40, and after the third convolution, the image resolution is 20*20. In the embodiment of the present application, the resolution of the second sample image in the second sample image set is 160*160. If there are images with other resolutions, please refer to Figure 2The change in resolution of the second sample image after each convolution is not described here because image processing by a convolution layer is a well-known technique to those skilled in the art.

[0108] The second sample image set is passed through Figure 2 After the convolutional layer, Figure 2 The result as Figure 3 Input, Figure 3 This is a schematic diagram of the acquisition process of the feature image set corresponding to the second sample image set. The result after the first convolution is used as input 1, the result after the second convolution is used as input 2, and the result after the third convolution is used as output 3. In order to extract the features of each second sample image in the second sample image set, input 1, input 2, and input 3 need to be processed by the pooling layer and the convolution layer respectively. The role of the pooling layer is to extract the features of the image and compress the thickness of the image. Figure 3 In the algorithm, the number of input channels is consistent with the number of output channels, which saves computer computing resources.

[0109] In the embodiment of the present application, the feature image output by the second channel is a fusion of the feature image collected by the first channel, and the feature image output by the third channel is a fusion of the feature images collected by the first channel and the second channel. The feature image collected by the pooling layer can be adjusted according to actual needs, which will not be elaborated here.

[0110] It should be further explained that through Figure 3 After the feature image set of the second sample image set is collected, the second sample image in the second sample image set can be mapped to the low-resolution image, such as Figure 4 FIG. 1 is a schematic diagram of the mapping process of the second sample image. Figure 4 The upper left 3*3 area of ​​image a is a human body image. The upper left 3*3 area of ​​image a is mapped to image b, and image b is 2*2. At this time, the human body image in image 4 has been converted from image form to digital form. The number represents the weight of the human body image in this area. The larger the number, the higher the possibility of the human body image in this area. 0 means that there is no human body image in this area. Further, image b can be mapped to image c. Therefore, by mapping image a to image c, it can be judged whether there is a human body image in image a based on image c.

[0111] get Figure 3 After the output results in , we need to get the prediction results, and use output 1, output 2, and output 3 as Figure 5 The input, in Figure 5 middle, Figure 3The output feature image will be convolved three times to obtain the classification branch, regression branch, and IOU prediction branch corresponding to the feature image. Finally, 9 sets of image data will be obtained. Each input corresponds to 3 types of image data. The classification branch is used to obtain the target object frame of the target object image, the regression branch is used to obtain the target object in the target object frame, and the IOU prediction branch is used to obtain the image corresponding to the minimum area containing the target object image. Among them, the feature images in the classification branch, regression branch, and IOU prediction branch are all in data form.

[0112] After segmenting the feature image set, in order to obtain the accuracy of the image processing model in detecting whether the staff member is wearing a seat belt, it is first necessary to divide the target object image in each second sample image in the second sample image set into a first target object image and a second target object image. The first target object image is the target object image with the larger size in the second sample image, and the second target object image is the target object image with the smaller size in the second sample image. The division process of the first target object image and the second target object image is as follows:

[0113] After determining the target object images in the second sample image, the image area of ​​each framed target object image is calculated, and the image areas are sorted from large to small. If the sequence number is an even number, the target object image in the first half of the sequence number is used as the first target object image, and the remaining target object images are used as the second target object image. If the sequence number is an odd number, the target object image in the first half after the sequence number is reduced by one is used as the first target object, and the remaining target object images are used as the second target object image.

[0114] After the first target object image and the second target object image are divided, the target object image needs to be divided into positive samples and negative samples. The positive sample is the image corresponding to the designated area near the center point of the target object image, and the negative sample is the image corresponding to the area outside the designated area of ​​the target object image. In the embodiment of the present application, the designated area is a 3*3 area near the center point of the target object image. Figure 7 The figure shows a schematic diagram of a positive sample. The 3*3 area near the center point of the target object image is a positive sample, and the area outside the positive sample in the image is a negative sample. The designated area can be adjusted according to the actual situation, so we will not elaborate on it here.

[0115] In the embodiment of the present application, the positive sample is the target object image, the negative sample is the background image of the target object image, the first target object image corresponds to the first positive sample and the first negative sample, and the second target object image corresponds to the second positive sample and the second negative sample.

[0116] Before using feature images of different resolutions to match the first positive sample, the first negative sample, the second positive sample, and the second negative sample, it is also necessary to determine the resolution of the feature images of the first positive sample, the first negative sample, the second positive sample, and the second negative sample. The resolution of the feature image is determined by the channel of the output feature image. The feature image of the third channel has low resolution and many fused image features. Since the feature image of the third channel is not clear enough, when the feature image of the third channel is used to detect the target object image, in order to more accurately detect the target object image, a larger target object image is required. Therefore, the first target object image is matched with the feature image of the third channel.

[0117] Since the feature image of the first channel has a high resolution, the feature image will be clearer. When the feature image of the first channel is used to detect the target object image, a target object image of a larger size is not required. Therefore, the second target object image is matched with the feature image of the first channel.

[0118] After determining that the first target object image is matched with the feature image of the third channel and the second target object image is matched with the feature image of the first channel, in order to obtain more accurate matching results, it is necessary to match the first positive sample with the feature image corresponding to the regression branch in the maximum resolution feature image to obtain image A, and match the first negative sample with the feature image corresponding to the classification branch in the maximum resolution feature image to obtain image a; match the second positive sample with the feature image corresponding to the regression branch in the minimum resolution feature image to obtain image B, and match the second negative sample with the feature image corresponding to the classification branch in the minimum resolution feature image to obtain image b.

[0119] After obtaining the A image and the a image, they need to be matched in the feature image corresponding to the IOU prediction branch of the maximum resolution, and the X image corresponding to the maximum IOU value is screened out. The feature image in the IOU prediction branch is the overlapping part of the real image and the feature image. Based on the same method, after obtaining the B image and the b image, they need to be matched in the feature image corresponding to the IOU prediction branch of the minimum resolution, and the Y image corresponding to the maximum IOU value is screened out. At this time, the first target object image will correspond to the A image, the a image, and the X image, and the second target object image will correspond to the B image, the b image, and the Y image. The target object image can be accurately locked according to the three matched images. Repeat the prediction training process described above to obtain the image processing model.

[0120] After obtaining the image processing model, the prediction value corresponding to the image processing model is calculated. The prediction value indicates the accuracy of the image processing model in locking the target object image. Then, the true value of the second sample image set is compared with the prediction value, and the second loss ratio corresponding to the feature image set of the second sample image set is calculated. The second loss ratio is fed back to the image processing model, and the second sample image set is processed according to the second loss ratio during the next model training, such as Figure 6 The figure shows the training diagram of the prediction network during the image processing model training process. Figure 6 In the embodiment, the first sample image set is input into the image processing model to obtain a first loss ratio. During the second model training, at least part of the first sample image set is expanded according to the first loss ratio to obtain a second sample image set. The second sample image set is trained by sequentially performing the steps in the above embodiment to obtain a second image processing model.

[0121] Through the method described above, the second sample image set is input into the prediction network for training, the feature image set corresponding to the second sample image set is extracted, and the feature image set is divided into three categories: classification branch, regression branch, and IOU prediction branch. The images in the feature image set are further divided, and the first target object image and the second target object image are matched with image feature sets of different resolutions respectively, ensuring the accuracy of the detection results corresponding to the image processing model.

[0122] According to the method described above, the image processing model is trained M times to obtain M loss ratios. After obtaining the M loss ratios, the M loss ratios are sorted from large to small, and the image processing model corresponding to the minimum loss ratio is screened out. The image processing model corresponding to the minimum loss ratio is used as the target image processing model.

[0123] Through the above method, the image processing model corresponding to the minimum loss ratio is selected as the target image processing model according to the size of the M loss ratios, ensuring that the target image processing model is the model with the highest accuracy among the image processing models, thereby improving the accuracy of the detection results of the target image processing model.

[0124] Example 2

[0125] Reference Figure 8 This application provides a behavior recognition method that can output an alarm message when abnormal behavior of a target object is detected. The implementation process of this method is as follows:

[0126] Step S81: Based on the trained target image processing model, behavior recognition is performed on the image to be processed containing the target object to determine whether the target object has abnormal behavior.

[0127] After obtaining the target image processing model, in the actual application process of the target image processing model, it is necessary to first obtain the video data of the staff, identify the target object in the current frame of the video, mark the target object with identification information, and detect whether the target object is wearing a seat belt. If abnormal behavior is detected in the target object of the current frame, an alarm message is output. In the second embodiment of the present application, the abnormal behavior is that the target object is not wearing a seat belt. In actual application, abnormal behavior includes the target object smoking, the target object driving and playing with a mobile phone, etc. Abnormal behavior is not elaborated too much here. After outputting the alarm message, if the identification information of the target object in the next frame of the current frame is inconsistent with the identification information of the target object in the current frame, an alarm message is output.

[0128] The above method prevents repeated output of alarm information to target objects that are not wearing seat belts, thereby improving the efficiency of detecting whether workers are wearing seat belts.

[0129] Example 3

[0130] Based on the same inventive concept, the present application also provides a training device for an image processing model. The seat belt detection device is used to implement the function of a training method for an image processing model. Figure 9 , the device comprises:

[0131] An iterative module 901 is configured to perform multiple iterative training on a first image processing model, and determine a target image processing model corresponding to the first image processing model based on the image processing models obtained from each iterative training and the loss ratio corresponding to each iterative training.

[0132] An expansion module 902 is configured to perform sample expansion processing on at least part of the i-th sample image set used in the i+1-th iterative training according to the i-th loss ratio corresponding to the i-th iterative training, to obtain at least one expanded sample;

[0133] The training module 903 is used to train the i-th image processing model obtained by the i-th iterative training based on the at least one extended sample and the first sample image set, and obtain the i+1-th image processing model corresponding to the i+1-th iterative training and the i+1-th loss ratio corresponding to the i+1-th iterative training.

[0134] In a possible design, the iteration module 901 is further configured to input the first sample image into a preset network for training to obtain a first image processing model.

[0135] In one possible design, the expansion module 902 is specifically used to determine whether the received i-th loss ratio exceeds a preset threshold value. If so, at least part of the i-th sample image in the i-th sample image set is scaled to obtain multiple scaled images, and scaled images of the same resolution in the multiple scaled images are spliced ​​to obtain at least one first extended sample, and the first extended sample is determined as the i-th sample image in the i-th sample image set. If not, at least part of the i-th sample image in the i-th sample image set is rotated to obtain multiple rotated images, and the pixels of the rotated images of the same size in the multiple rotated images are added according to the weight value of the image to obtain at least one second extended sample, and the second extended sample is determined as the i-th sample image in the i-th sample image set.

[0136] In one possible design, the expansion module 902 is further used to convert the at least one extended sample in the RGB space and each image in the i-th sample image set into the HIS space, decompose each image into a high-frequency component and a low-frequency component according to brightness, process the high-frequency component with linear weighted enhancement, and process the low-frequency component with histogram equalization, fuse the high-frequency component processed with linear weighted enhancement and the low-frequency component processed with histogram equalization in each image, and invert each fused image from the HIS space to the RGB space.

[0137] In one possible design, the training module 903 is specifically used to input the at least one extended sample and the i-th sample image set into the prediction network for training, obtain the image feature set of the i+1-th sample image set, perform prediction training on the i+1-th sample image set based on the image feature set, and obtain the i+1-th loss ratio.

[0138] In one possible design, the training module 903 is also used to obtain the target object image in the i+1th sample image set, detect whether the image area of ​​the target object image exceeds a preset area, and if so, use the target object image exceeding the preset area as the first target object image; if not, use the target object image below the preset area as the second target object image, and perform prediction training on the first target object image and the second target object image based on the feature image set according to preset rules.

[0139] In one possible design, the training module 903 is also used to extract a first positive sample and a first negative sample corresponding to the first target object image and a second positive sample and a second negative sample corresponding to the second target object image, respectively, wherein the positive sample is an image corresponding to a specified area in the target object image, and the negative sample is an image corresponding to an area outside the specified area in the target object image. When the target object image is the first target object image, the first positive sample and the first negative sample are predicted using a feature image of a first resolution; when the target object image is the second target object image, the second positive sample and the second negative sample are predicted using a feature image of a second resolution.

[0140] Example 4

[0141] Based on the same inventive concept, the embodiment of the present application also provides a device for behavior recognition, and the seat belt detection device is used to realize the function of a behavior recognition method, referring to Figure 10 , the device comprises:

[0142] The recognition module 1001 is used to perform behavior recognition on the image to be processed containing the target object based on the trained target image processing model to determine whether the target object has abnormal behavior.

[0143] In one possible design, the identification module 1001 is specifically used to determine that the target object in the current frame has abnormal behavior, and then output an alarm message; and to determine that the identification information of the target object in the next frame of the current frame is inconsistent with the identification information of the target object in the current frame, and then output an alarm message.

[0144] Example 5

[0145] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, which can realize the functions of the aforementioned image processing model training device and the function of a behavior recognition device, reference Figure 11 , the electronic device includes:

[0146] At least one processor 1101, and a memory 1102 connected to the at least one processor 1101. The specific connection medium between the processor 1101 and the memory 1102 is not limited in the embodiment of the present application. Figure 11 In the example, the processor 1101 and the memory 1102 are connected via the bus 1100. Figure 11 The bus 1100 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 11The diagram is represented by only one thick line, but this does not mean that there is only one bus or one type of bus. Alternatively, the processor 1101 may also be referred to as a controller, without limitation to the name.

[0147] In the embodiment of the present application, the memory 1102 stores instructions that can be executed by at least one processor 1101. The at least one processor 1101 can execute the training method of an image processing model and a behavior recognition method discussed above by executing the instructions stored in the memory 402. The processor 1101 can implement Figure 8 as well as Figure 9 The functions of each module in the device shown.

[0148] Among them, the processor 1101 is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device as a whole by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102, the various functions of the device and processing data.

[0149] In one possible design, processor 1101 may include one or more processing units. Processor 1101 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1101. In some embodiments, processor 1101 and memory 1102 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0150] Processor 1101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the training method of an image processing model and a method of behavior recognition disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0151] Memory 1102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1102 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. Memory 1102 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1102 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0152] By designing and programming the processor 1101, the codes corresponding to the training method of an image processing model and the method of behavior recognition described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 1 The training method steps and execution steps of an image processing model in the embodiment shown Figure 8 The steps of a behavior recognition method in the embodiment shown are as follows: How to design and program the processor 1101 is a technique well known to those skilled in the art and will not be described in detail here.

[0153] Based on the same inventive concept, an embodiment of the present application also provides a storage medium, which stores computer instructions. When the computer instructions are run on a computer, the computer executes an image processing model training method and a behavior recognition method discussed above.

[0154] In some possible embodiments, the present application provides various aspects of a method for training an image processing model and a method for behavior recognition, which can also be implemented in the form of a program product, which includes program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the method for training an image processing model and a method for behavior recognition according to various exemplary embodiments of the present application described above in this specification.

[0155] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of complete hardware embodiment, complete software embodiment, or the embodiment in conjunction with software and hardware. Moreover, the application can adopt the form of the computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CDROM, optical storage, etc.) that include computer-usable program code.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0159] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A training method for an image processing model, characterized in that: include: Performing multiple iterative training on a first image processing model, and determining a target image processing model corresponding to the first image processing model based on an image processing model obtained by each iterative training in the multiple iterative training and a loss ratio corresponding to each iterative training, wherein the i+1th iterative training in the multiple iterative training includes: Determine whether the proportion of i-th loss corresponding to the received i-th iterative training exceeds a preset threshold; if so, scale at least part of the i-th sample image in the i-th sample image set used in the (i+1)-th iterative training to obtain a plurality of scaled images, and splice scaled images of the same resolution in the plurality of scaled images to obtain at least one first extended sample, and determine the first extended sample as the i-th sample image in the i-th sample image set; if not, rotate at least part of the i-th sample image in the i-th sample image set to obtain a plurality of rotated images, and add pixels of the rotated images of the same size in the plurality of rotated images according to image weight values ​​to obtain at least one second extended sample, and determine the second extended sample as the i-th sample image in the i-th sample image set; Based on at least one extended sample and the i-th sample image set, the i-th image processing model obtained by the i-th iterative training is trained to obtain the i+1-th image processing model corresponding to the i+1-th iterative training and the i+1-th loss ratio corresponding to the i+1-th iterative training, where i is a positive integer; and the extended sample is the first extended sample or the second extended sample.

2. The method according to claim 1, wherein Determining a target image processing model corresponding to the first image processing model based on an image processing model obtained by each iterative training in multiple iterative trainings and a loss ratio corresponding to each iterative training, including: Obtain M image processing models trained through multiple iterations and M loss ratios, where M is a positive integer; The image processing model corresponding to the minimum loss ratio is used as the target image processing model.

3. The method according to claim 1, wherein The process of obtaining the first image processing model includes: The first sample image is input into a preset network for training to obtain a first image processing model.

4. The method according to claim 1, wherein Before training the i-th image processing model obtained by the i-th iterative training, the following steps are included: Converting the at least one extended sample in the RGB space and each image in the i-th sample image set into the HIS space; Decomposing each image into a high-frequency component and a low-frequency component according to brightness, the high-frequency component is processed by linear weighted enhancement, and the low-frequency component is processed by histogram equalization; The high-frequency components processed by linear weighted enhancement and the low-frequency components processed by histogram equalization in each image are fused, and each fused image is inversely transformed from the HIS space to the RGB space.

5. The method according to claim 1, wherein Based on the at least one extended sample and the i-th sample image set, the i-th image processing model obtained by the i-th iterative training is trained to obtain the i+1-th image processing model corresponding to the i+1-th iterative training and the i+1-th loss ratio corresponding to the i+1-th iterative training, including: Inputting the at least one extended sample and the i-th sample image set into a prediction network for training to obtain an image feature set of the i+1-th sample image set, wherein the images in the image feature set have different resolutions; Obtain a target object image in the i+1th sample image set, and detect whether the image area of ​​the target object image exceeds a preset area; if so, use the target object image exceeding the preset area as the first target object image; if not, use the target object image below the preset area as the second target object image; perform prediction training on the first target object image and the second target object image based on the image feature set according to preset rules to obtain the i+1th loss ratio.

6. The method according to claim 5, wherein Performing prediction training on the first target object image and the second target object image based on the image feature set according to a preset rule, including: Extracting a first positive sample and a first negative sample corresponding to the first target object image and a second positive sample and a second negative sample corresponding to the second target object image, respectively, wherein the positive sample is an image corresponding to a specified area in the target object image, and the negative sample is an image corresponding to an area outside the specified area in the target object image; When the target object image is a first target object image, predicting the first positive sample and the first negative sample using a feature image of a first resolution; When the target object image is a second target object image, the second positive sample and the second negative sample are predicted using a feature image of a second resolution.

7. A method for behavior recognition, characterized in that: include: Based on the trained target image processing model, behavior recognition is performed on the image to be processed containing the target object to determine whether the target object has abnormal behavior, wherein the abnormal behavior is that the target object is not wearing a seat belt, and the target image processing model is obtained by training the first image processing model based on the method described in any one of claims 1-6.

8. The method according to claim 7, wherein Performing behavior recognition on the image to be processed containing the target object based on the trained target image processing model to determine whether the target object has abnormal behavior, including: If it is determined that the target object in the current frame has abnormal behavior, an alarm message is output; If it is determined that the identification information of the target object in the next frame of the current frame is inconsistent with the identification information of the target object in the current frame, an alarm message is output.

9. A training device for an image processing model, characterized in that: The device comprises: an iterative module, configured to perform multiple iterative training on the first image processing model, and determine a target image processing model corresponding to the first image processing model based on the image processing model obtained from each iterative training and the loss ratio corresponding to each iterative training in the multiple iterative training; an expansion module, configured to determine whether the received i-th loss ratio exceeds a preset threshold; if so, scaling at least part of the i-th sample image in the i-th sample image set to obtain a plurality of scaled images, and splicing scaled images of the same resolution in the plurality of scaled images to obtain at least one first extended sample, and determining the first extended sample as the i-th sample image in the i-th sample image set; if not, rotating at least part of the i-th sample image in the i-th sample image set to obtain a plurality of rotated images, and adding pixels of the rotated images of the same size in the plurality of rotated images according to image weight values ​​to obtain at least one second extended sample, and determining the second extended sample as the i-th sample image in the i-th sample image set; A training module is used to train the i-th image processing model obtained by the i-th iterative training based on at least one extended sample and a first sample image set, to obtain the i+1-th image processing model corresponding to the i+1-th iterative training and the i+1-th loss ratio corresponding to the i+1-th iterative training, wherein i is a positive integer; and the extended sample is the first extended sample or the second extended sample.

10. A device for behavior recognition, characterized in that: The device comprises: An identification module is used to perform behavior recognition on an image to be processed containing a target object based on a trained target image processing model to determine whether the target object has abnormal behavior, wherein the abnormal behavior is that the target object is not wearing a seat belt, and the target image processing model is obtained by training a first image processing model based on the method described in any one of claims 1-6.

11. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 8 when executing the computer program stored in the memory.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 8 are implemented.

Images