Model training method, apparatus and device

Abstract

This application provides a model training method, apparatus, and device. The method includes: acquiring a first dataset, which includes multiple first sample images and image features of each first sample image; training a model using the first dataset to obtain a residual network, the residual network including at least one convolutional block and at least one residual block; acquiring a second dataset, which includes multiple second sample images and a labeling result for each second sample image, the labeling result indicating whether an object in the second sample image exhibits a preset behavior; and training a target model using the second dataset to obtain a target model, the target model including a residual network and a multi-feature fusion network, the multi-feature fusion network including a spatial pyramid pooling layer and a regularized convolutional layer, the target model being used to identify whether an object in an image exhibits a preset behavior. This improves the accuracy of determining whether mobile phone usage behavior exists.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, apparatus, and device. Background Technology

[0002] In industrial manufacturing settings (e.g., factories, gas stations, workshops), the use of mobile phones poses security risks. To ensure safety in industrial manufacturing, it is necessary to monitor the use of mobile phones in these settings in real time.

[0003] In related technologies, the presence of mobile phone use can be determined as follows: images of industrial manufacturing scenes are acquired through surveillance video; hand contour features are extracted from the images using image processing algorithms; classification algorithms are used to categorize the hand contour features to obtain type information corresponding to the hand contour features; and based on the type information, it is determined whether a mobile phone use action occurred. However, this process only extracts and analyzes features from the hand images and cannot identify whether mobile phone use occurred in special scenarios (e.g., low lighting, off-center shooting positions), resulting in low accuracy in determining whether mobile phone use occurred. Summary of the Invention

[0004] This application provides a model training method, apparatus, and device to address the problem of low accuracy in determining whether mobile phone usage behavior exists.

[0005] In a first aspect, embodiments of this application provide a model training method, including:

[0006] Obtain a first data set, which includes multiple first sample images and image features of each first sample image;

[0007] The model is trained using a first dataset to obtain a residual network, which includes at least one convolutional block and at least one residual block, and each convolutional block and residual block is followed by an average pooling layer and a convolutional layer.

[0008] Obtain a second data set, which includes multiple second sample images and a labeling result for each second sample image. The labeling result is used to indicate whether an object in the second sample image exhibits a preset behavior.

[0009] The target model is obtained by training the model using the second dataset. The target model includes the residual network and the multi-feature fusion network. The multi-feature fusion network includes a spatial pyramid pooling layer and a regularized convolutional layer. The target model is used to identify whether an object in an image exhibits the preset behavior.

[0010] In one possible implementation, the target model is obtained by training the model using a second dataset, including:

[0011] In the second dataset, determine the training dataset and the validation dataset;

[0012] The model is trained using the training dataset to obtain an initial model;

[0013] The initial model is trained using the validation dataset to obtain the target model.

[0014] In one possible implementation, model training is performed using the training dataset to obtain an initial model, including:

[0015] The model parameters of the first model are initialized using the model parameters of the residual network to obtain the initialized first model;

[0016] The training dataset is subjected to multiple image enhancement processes to obtain multiple enhanced datasets;

[0017] The initial model is obtained by iteratively training the first model after initialization using the training dataset and the multiple augmentation datasets respectively.

[0018] In one possible implementation, the initialized first model is trained iteratively multiple times using the training dataset and the plurality of augmented datasets to obtain the initial model, including:

[0019] The first initialized model is trained in the first iteration using the training dataset to obtain the first intermediate model;

[0020] The i-th intermediate model is trained for the (i+1)th iteration using the i-th augmented dataset to obtain the (i+1)-th intermediate model. The i-th intermediate model is obtained by taking the values ​​1, 2, 3, ..., until the i-th intermediate model converges. When i is greater than or equal to N, the i-th intermediate model is determined as the initial model, where N is an integer greater than 1.

[0021] In one possible implementation, the labeling result includes the labeling position of the bounding box; the i-th intermediate model is trained for the (i+1)-th iteration using the i-th augmented dataset to obtain the (i+1)-th intermediate model, including:

[0022] The i-th intermediate model processes the sample images in the i-th augmented dataset to obtain prediction results, which include the predicted positions of the prediction boxes.

[0023] The marked area of ​​the marked box is determined based on the marked position, and the predicted area of ​​the predicted box is determined based on the predicted position;

[0024] The loss value is determined based on the predicted location, the predicted area, the marked location, the marked area, and a preset loss function;

[0025] The model parameters of the i-th intermediate model are updated based on the loss value to obtain the (i+1)-th intermediate model.

[0026] In one possible implementation, for any one of the multiple image enhancement processes; image enhancement processing is performed on the training dataset to obtain an enhanced dataset, including:

[0027] The third sample image is obtained by performing image enhancement processing on the second sample image in the training dataset. The image enhancement processing includes at least one of the following: scaling, flipping, translation, rotation, grayscale filling, mean normalization, variance standardization, color transformation, image blending, and random multi-scale operation.

[0028] For any second sample image, the labeling result corresponding to the second sample image is adaptively adjusted to obtain the labeling result corresponding to the third sample image.

[0029] In one possible implementation, the labeling result includes a first labeling result and a second labeling result, wherein,

[0030] The first labeling result includes a first image type, which indicates that the object in the second sample image does not exhibit the preset behavior; or,

[0031] The second labeling result includes a second image type and the labeling position of the image region corresponding to the preset behavior in the second sample image. The second image type is used to indicate that the object in the second sample image has the preset behavior.

[0032] In one possible implementation, the labeling result corresponding to the second sample image is adaptively adjusted to obtain the labeling result corresponding to the third sample image, including:

[0033] If the labeling result corresponding to the second sample image is the first labeling result, then the labeling result corresponding to the third sample image is determined to be the first labeling result;

[0034] If the labeling result corresponding to the second sample image is the second labeling result, then the labeling position in the second labeling result is updated according to the image enhancement process to obtain the updated labeling position, and the labeling result corresponding to the third sample image is determined to include the updated labeling position and the second image type.

[0035] In one possible implementation, the method further includes:

[0036] Get the first video to be processed;

[0037] Determine image frames in the first video;

[0038] The image frame is processed using the target model to obtain the prediction result;

[0039] The prediction result includes a first image type, or the prediction result includes a second image type and the predicted position of the image region corresponding to the preset behavior in the image frame. The first image type is used to indicate that the object in the image frame does not have the preset behavior, and the second image type is used to indicate that the object in the image frame has the preset behavior.

[0040] Secondly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0041] The first acquisition module is used to acquire a first data set, which includes multiple first sample images and image features of each first sample image;

[0042] The first training module is used to train the model using a first dataset to obtain a residual network. The residual network includes at least one convolutional block and at least one residual block, and each convolutional block and the residual block is followed by an average pooling layer and a convolutional layer.

[0043] The second acquisition module is used to acquire a second data set, which includes multiple second sample images and a labeling result for each second sample image. The labeling result is used to indicate whether an object in the second sample image has a preset behavior.

[0044] The second training module is used to train the model using the second dataset to obtain the target model. The target model includes the residual network and the multi-feature fusion network. The multi-feature fusion network includes a spatial pyramid pooling layer and a regularized convolutional layer. The target model is used to identify whether an object in the image exhibits the preset behavior.

[0045] In one possible implementation, the second training module is specifically used for:

[0046] In the second dataset, determine the training dataset and the validation dataset;

[0047] The model is trained using the training dataset to obtain an initial model;

[0048] The initial model is trained using the validation dataset to obtain the target model.

[0049] In one possible implementation, the second training module is specifically used for:

[0050] The model parameters of the first model are initialized using the model parameters of the residual network to obtain the initialized first model;

[0051] The training dataset is subjected to multiple image enhancement processes to obtain multiple enhanced datasets;

[0052] The initial model is obtained by iteratively training the first model after initialization using the training dataset and the multiple augmentation datasets respectively.

[0053] In one possible implementation, the second training module is specifically used for:

[0054] The first initialized model is trained in the first iteration using the training dataset to obtain the first intermediate model;

[0055] The i-th intermediate model is trained for the (i+1)th iteration using the i-th augmented dataset to obtain the (i+1)-th intermediate model. The i-th intermediate model is obtained by taking the values ​​1, 2, 3, ..., until the i-th intermediate model converges. When i is greater than or equal to N, the i-th intermediate model is determined as the initial model, where N is an integer greater than 1.

[0056] In one possible implementation, the second training module is specifically used for:

[0057] The i-th intermediate model processes the sample images in the i-th augmented dataset to obtain prediction results, which include the predicted positions of the prediction boxes.

[0058] The marked area of ​​the marked box is determined based on the marked position, and the predicted area of ​​the predicted box is determined based on the predicted position;

[0059] The loss value is determined based on the predicted location, the predicted area, the marked location, the marked area, and a preset loss function;

[0060] The model parameters of the i-th intermediate model are updated based on the loss value to obtain the (i+1)-th intermediate model.

[0061] In one possible implementation, the second training module is specifically used for:

[0062] The third sample image is obtained by performing image enhancement processing on the second sample image in the training dataset. The image enhancement processing includes at least one of the following: scaling, flipping, translation, rotation, grayscale filling, mean normalization, variance standardization, color transformation, image blending, and random multi-scale operation.

[0063] For any second sample image, the labeling result corresponding to the second sample image is adaptively adjusted to obtain the labeling result corresponding to the third sample image.

[0064] In one possible implementation, the second training module is specifically used for:

[0065] The first labeling result includes a first image type, which indicates that the object in the second sample image does not exhibit the preset behavior; or,

[0066] The second labeling result includes a second image type and the labeling position of the image region corresponding to the preset behavior in the second sample image. The second image type is used to indicate that the object in the second sample image has the preset behavior.

[0067] In one possible implementation, the second training module is specifically used for:

[0068] If the labeling result corresponding to the second sample image is the first labeling result, then the labeling result corresponding to the third sample image is determined to be the first labeling result;

[0069] If the labeling result corresponding to the second sample image is the second labeling result, then the labeling position in the second labeling result is updated according to the image enhancement process to obtain the updated labeling position, and the labeling result corresponding to the third sample image is determined to include the updated labeling position and the second image type.

[0070] In one possible implementation, the device further includes a third acquisition module, a determination module, and a processing module.

[0071] The third acquisition module is used for:

[0072] Get the first video to be processed.

[0073] The determining module is used for:

[0074] Identify image frames in the first video.

[0075] The processing module is used for:

[0076] The image frame is processed by the target model to obtain the prediction result.

[0077] The prediction result includes a first image type, or the prediction result includes a second image type and the predicted position of the image region corresponding to the preset behavior in the image frame. The first image type is used to indicate that the object in the image frame does not have the preset behavior, and the second image type is used to indicate that the object in the image frame has the preset behavior.

[0078] Thirdly, this application provides a chip on which a computer program is stored, and when the computer program is executed by the chip, it implements the method described in any of the first aspects.

[0079] Fourthly, this application provides a chip module on which a computer program is stored, and when the computer program is executed by the chip module, it implements the method described in any of the first aspects.

[0080] Fifthly, embodiments of this application provide a model training device, comprising:

[0081] At least one processor; and

[0082] A memory communicatively connected to the at least one processor; wherein,

[0083] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in any of the first aspects.

[0084] In a sixth aspect, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in any one of the first aspects.

[0085] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any one of the first aspects.

[0086] The model training method, apparatus, and device provided in this application acquire sample images and their corresponding image features. A residual network is obtained through model training using these sample images and their corresponding image features. The model containing the residual network is then trained based on the sample images and their labeling results to obtain a target model. This target model is then used to determine whether an object in an image is using a mobile phone. In this process, the target model can determine the labeling result of the image based on the image and its corresponding image features. The target model can extract features from the entire image, not just hand contour features, and can determine whether an object in the image is using a mobile phone in different scenarios based on the labeling results, thus improving the accuracy of determining whether mobile phone use has occurred. Attached Figure Description

[0087] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application;

[0088] Figure 2 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0089] Figure 3 This is a schematic diagram of the residual network structure provided in an embodiment of this application;

[0090] Figure 4 A schematic diagram of the second sample image provided in the embodiments of this application;

[0091] Figure 5 A flowchart illustrating another model training method provided in an embodiment of this application;

[0092] Figure 6 A schematic diagram illustrating the process of determining the marker position provided in an embodiment of this application;

[0093] Figure 7 A flowchart illustrating a method for determining whether an object exhibits a preset behavior, as provided in an embodiment of this application.

[0094] Figure 8 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0095] Figure 9 This is a schematic diagram of another model training device provided in an embodiment of this application;

[0096] Figure 10 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application. Detailed Implementation

[0097] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0098] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0099] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 The system includes a detection device 101 and a user 102. The detection device 101 can be a smart terminal device, such as a smart camera. The detection device 101 can detect whether the user 102 is using a mobile phone at a certain moment. The detection device 101 can acquire an image within the detection range at a certain moment and extract the hand contour feature information of the user 102 from the image. Based on the hand contour feature information of the user 102, it is determined whether the user 102 is using a mobile phone.

[0100] In related technologies, the presence of mobile phone use can be determined as follows: images of industrial manufacturing scenes are acquired through surveillance video; image processing algorithms are used to extract the hand contour features of objects in the images; classification algorithms are used to classify the hand contour features to obtain type information corresponding to the hand contour features; and based on the type information, it is determined whether a mobile phone use action has occurred. However, this process only extracts and analyzes features from hand images and cannot identify whether mobile phone use occurs in special scenarios (e.g., low lighting, off-center shooting positions), resulting in low accuracy in determining whether mobile phone use has occurred.

[0101] In this embodiment, sample images and their corresponding image features are obtained. A residual network is then trained using these features. Based on the sample images and their labeling results, the model containing the residual network is trained to obtain a target model. This target model is then used to determine whether an object in the image is using a mobile phone. In this process, the target model can determine the image's labeling based on the image and its corresponding image features. The target model can extract features from the entire image, not just hand contour features, and can determine whether an object in the image is using a mobile phone in different scenarios based on the labeling results, thus improving the accuracy of determining whether mobile phone use has occurred.

[0102] The method described in this application will now be illustrated through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; identical or similar content will not be repeated in different embodiments.

[0103] Figure 2 This is a schematic flowchart illustrating a model training method provided in an embodiment of this application. Please refer to [link / reference]. Figure 2 The method may include:

[0104] S201, Obtain the first data set.

[0105] The execution entity in this application embodiment can be a model training device, or a chip, chip module, or model training apparatus disposed within the model training device. The model training apparatus can be implemented through software or a combination of software and hardware. The model training device can be a computer.

[0106] The first dataset includes multiple first sample images and the image features of each first sample image.

[0107] The first dataset can be obtained from the ImageNet dataset. The image features of the first sample images can include color features, texture features, and shape features, etc.

[0108] S202. Train the model using the first dataset to obtain the residual network.

[0109] The residual network includes at least one convolutional block and at least one residual block, and each convolutional block and residual block is followed by an average pooling layer and a convolutional layer.

[0110] The residual network can be a Deep Residual Network (ResNet-50)-D, which is an updated version of ResNet50. ResNet50 includes at least one convolutional block and at least one residual block. Average pooling layers and convolutional layers can be inserted after each convolutional block and residual block, respectively, to obtain ResNet50-D.

[0111] Below, in conjunction with Figure 3 The structure of the residual network is explained. Figure 3 This is a schematic diagram of the residual network provided in an embodiment of this application. Please refer to [link / reference]. Figure 3 The network includes a residual network 301. The residual network 301 comprises one convolutional block (Conv Block) and multiple residual blocks. These residual blocks are divided into four groups, each group extracting image features within a different receptive field. The first group contains one residual block with a receptive field size of (208, 208, 64). The second group contains two residual blocks with receptive field sizes of (104, 104, 128). The third group contains eight residual blocks with receptive field sizes of (26, 26, 512). The fourth group contains four residual blocks with receptive field sizes of (13, 13, 1024). After the convolutional block and each residual block, a mean pooling layer and a convolutional layer are inserted.

[0112] Inserting an average pooling layer and a convolutional layer after each convolutional block and residual block can preserve the features extracted from each convolutional block and residual block, preventing the loss of certain image features.

[0113] The model is trained using the first dataset to determine the model parameters of the residual network. These residual network parameters are then used as the model parameters of the initialized first model.

[0114] S203, Obtain the second data set.

[0115] The second dataset includes multiple second sample images and the labeling results for each second sample image. The labeling results are used to indicate whether the objects in the second sample images exhibit a pre-defined behavior.

[0116] Secondary sample images can be obtained by taking photos or downloading images from the internet. The scene of the secondary sample image can be an industrial manufacturing scene, and the objects in the secondary sample image can include portraits of people using mobile phones or other actions. Industrial manufacturing scenes can include factory workshops, factory kitchens, construction sites, warehouses, etc.

[0117] The behavior of objects in the second sample image can be represented numerically. For example, the preset behavior could be using a mobile phone. If an object in the second sample image exhibits the behavior of using a mobile phone, it is represented by 1; if not, it is represented by 0. Based on the numerical value corresponding to the labeling result, it is determined whether the object in the second sample image exhibits the preset behavior.

[0118] Below, in conjunction with Figure 4 The second sample image and its labeling results are explained. Figure 4 This is a schematic diagram of a second sample image provided for an embodiment of this application. Please refer to [link / reference]. Figure 4 The second sample image 401 contains a human figure. Assuming the preset behavior is using a mobile phone, the presence of a mobile phone user in the second sample image 401 is represented by 1, and the absence of such behavior is represented by 0. Therefore, the number corresponding to the labeling result of the second sample image 401 can be 1, indicating that the person in the second sample image 401 is using a mobile phone.

[0119] S203. The target model is obtained by training the model using the second dataset.

[0120] The target model includes a residual network and a multi-feature fusion network. The multi-feature fusion network includes a spatial pyramid pooling layer and a regularized convolutional layer. The target model is used to identify whether objects in an image exhibit a pre-defined behavior.

[0121] Spatial pyramid pooling layers can fuse features extracted by residual networks from different receptive fields. Regularized convolutional layers can randomly remove some feature information, allowing the target model to randomly learn the remaining features multiple times, thus improving the generalization ability of the target model.

[0122] The target model can be obtained as follows: determine the training dataset and validation dataset in the second dataset; train the model using the training dataset to obtain the initial model; train the initial model using the validation dataset to obtain the target model.

[0123] The model training method provided in this application involves obtaining a first dataset; training a model using the first dataset to obtain a residual network; obtaining a second dataset; and training a model using the second dataset to obtain a target model. In this process, the target model includes a residual network and a multi-feature fusion network. The average pooling layer and convolutional layer in the residual network can retain and extract as much feature information as possible from the sample images. The spatial pyramid pooling layer of the multi-feature fusion network can improve the generalization ability of the target model, enabling it to identify whether objects in images are using mobile phones in various scenarios, thus improving the accuracy of determining whether objects are using mobile phones through the target model.

[0124] Based on any of the above embodiments, the following, in conjunction with Figure 5 The detailed process of model training is explained.

[0125] Figure 5 This is a flowchart illustrating another model training method provided in an embodiment of this application. Please refer to... Figure 5 The method may include:

[0126] S501, Obtain the first data set.

[0127] It should be noted that the execution process of S501 can be found in the execution process of S201, and will not be repeated here.

[0128] S502. Train the model using the first dataset to obtain the residual network.

[0129] The residual network is trained using the first dataset to obtain its weight file. The weight file includes the structure and model parameters of the residual network.

[0130] S503. Initialize the model parameters of the first model using the model parameters of the residual network to obtain the initialized first model.

[0131] The first model can be based on the You Only Look Once (YOLOv3) model, using a ResNet50-D residual network to replace the main network of the YOLOv3 model. The model parameters of the residual network are used as the initial model parameters for the first model.

[0132] S504, Obtain the second data set.

[0133] The second dataset includes multiple second sample images and the labeling results for each second sample image.

[0134] The labeling results include a first labeling result and a second labeling result. The first labeling result includes a first image type, which indicates that the object in the second sample image does not exhibit the preset behavior. The second labeling result includes a second image type and the labeling position of the image region corresponding to the preset behavior in the second sample image. The second image type indicates that the object in the second sample image exhibits the preset behavior.

[0135] For any given second sample image, it can be labeled as follows: acquire the second sample image; determine whether the object in the second sample image has a preset behavior; if the object in the second sample image has a preset behavior, extract the image frame of the object's behavior using a labeling tool and label it with a label box; generate a file in VOC format from the labeled second sample image and save it in the database.

[0136] The default behavior can be the behavior of using a mobile phone. The annotation tool can be the graphic image annotation tool Labellmg.

[0137] The marked position can be represented by coordinates. Below, we will combine... Figure 6 The location of the markers in the second sample image is explained. Figure 6 This is a schematic diagram illustrating the process of determining the marker position as provided in an embodiment of this application. Please refer to... Figure 6 The dataset includes a second sample image 601 and a labeled sample image 602. Assuming the preset behavior can be mobile phone use, it can be determined that the object in the second sample image 601 exhibits mobile phone use behavior. In this case, the labeling result of the second sample image 601 includes the second image type and the labeled position of the image region corresponding to the mobile phone use behavior within the second sample image. When determining the labeling position, a two-dimensional coordinate system is established with the lower left corner of the second sample image 601 as the origin O, the horizontal x-axis as the origin O, and the vertical y-axis as the origin O, resulting in the labeled sample image 602. In the labeled sample image 602, the image region corresponding to the mobile phone use behavior is labeled using a bounding box A. The center point of the labeled region and the side length of the bounding box A are determined. The side length of the bounding box includes its width and height. The coordinates of the center point and the side length (x1, y1, w, h) of the bounding box are used to determine the labeling position in the second sample image 601.

[0138] For example, the preset behavior could be the act of using a mobile phone. The first image type can be represented by 0, indicating that the object in the second sample image does not exhibit the behavior of using a mobile phone. The second image type can be represented by 1, indicating that the object in the second sample image exhibits the behavior of using a mobile phone.

[0139] For example, the specific labeling results can be shown in Table 1:

[0140] Table 1

[0141] Tagging results First labeling result Second labeling result Image type 0 1 Mark location — (x1, y1, w, h)

[0142] S505. Determine the training dataset and validation dataset in the second dataset.

[0143] The training dataset is used to train the first model, and the validation dataset is used to validate the accuracy of the first model.

[0144] The second training data can be randomly divided into a training dataset and a validation dataset according to a preset ratio. For example, the second dataset may contain 100 sets of data, each set including one second sample image and its labeling result. If the preset ratio in the second dataset is 8:2, then 80 sets of data in the second dataset will be designated as the training dataset, and 20 sets as the validation dataset.

[0145] S506. The first model after initialization is trained for the first iteration using the training dataset to obtain the first intermediate model.

[0146] During the first iteration of training, the training dataset without image augmentation was used for iterative training.

[0147] S507. Initialize i to 1.

[0148] S508. Perform the i-th image enhancement process on the training dataset to obtain the i-th enhanced dataset.

[0149] For any one of the multiple image enhancement processes, the second sample image in the training dataset is enhanced to obtain the third sample image. The image enhancement processes include at least one of the following: scaling, flipping, translation, rotation, grayscale filling, mean normalization, variance standardization, color transformation, image blending, and random multi-scale operation. For any second sample image, the labeling result corresponding to the second sample image is adaptively adjusted to obtain the labeling result corresponding to the third sample image.

[0150] The labeling result corresponding to the third sample image can be obtained in the following way: if the labeling result corresponding to the second sample image is the first labeling result, then the labeling result corresponding to the third sample image is determined to be the first labeling result; if the labeling result corresponding to the second sample image is the second labeling result, then the labeling position in the second labeling result is updated according to the image enhancement processing to obtain the updated labeling position, and the labeling result corresponding to the third sample image is determined to include the updated labeling position and the second image type.

[0151] For example, the second sample image is image A, and the labeling result corresponding to image A is the first labeling result. After image enhancement processing is performed on image A, the third sample image A1 is obtained. Since the labeling result corresponding to image A is the first labeling result, it can be determined that the labeling result corresponding to the third sample image A1 is the first labeling result.

[0152] For example, the second sample image is image B, and the labeling result corresponding to image B is the second labeling result, with the labeling position (x1, y1, w1, h1). After image enhancement processing is performed on image B, the third sample image B1 is obtained. Since the labeling result corresponding to image B is the second labeling result, the labeling position (x2, y2, w2, h2) in the second labeling result is updated according to the image enhancement processing, resulting in the updated labeling position (x2, y2). Determining the labeling result corresponding to the third sample image B1 includes the updated labeling position (x2, y2) and the second image type.

[0153] S509. The sample images in the i-th augmented dataset are processed by the i-th intermediate model to obtain the prediction results.

[0154] i takes values ​​of 1, 2, 3, ..., N in sequence. The prediction result includes the predicted position of the bounding box. The (N+1)th intermediate model is determined as the initial model.

[0155] The predicted location can be represented using two-dimensional coordinates.

[0156] The prediction result can be expressed by Formula 1:

[0157] (x, y, w, h, Score_confidence)

[0158] Where x is the offset between the x-coordinates of the predicted position and the labeled position; y is the offset between the y-coordinates of the predicted position and the labeled position; w is the ratio of the width of the predicted bounding box to the width of the sample image; h is the ratio of the height of the predicted bounding box to the height of the sample image; and Score_confidence is the confidence level of the preset behavior.

[0159] The confidence level of the preset behavior can be determined using Formula 2:

[0160]

[0161] Wherein, Pr(Object) indicates whether there is a preset behavior in the sample image, with 1 indicating the presence of the preset behavior and 0 indicating the absence of the preset behavior; `truth` represents the overlap ratio between the predicted bounding box and the labeled bounding box; `truth` represents the area of ​​the labeled bounding box; `pred` represents the area of ​​the predicted bounding box.

[0162] S510. Determine the marked area of ​​the marked box based on the marked position, and determine the predicted area of ​​the predicted box based on the predicted position.

[0163] The side length of the marker box can be determined based on the marker position, and the product of the two side lengths of the marker box is determined as the marker area of ​​the marker box.

[0164] The side length of the prediction box can be determined based on the predicted location, and the product of the two side lengths of the prediction box is determined as the predicted area of ​​the prediction box.

[0165] S511. Determine the loss value based on the predicted location, predicted area, marked location, marked area, and the preset loss function.

[0166] The loss function can be the GIoU loss function.

[0167] The loss value can be determined using Formula 3:

[0168]

[0169] Among them, L GIoU A is the loss value; B is the area of ​​the bounding box; C is the area of ​​the smallest rectangle parallel to the coordinate axis that encloses the two rectangular regions A and B.

[0170] S512. Determine whether the loss value no longer changes within the most recent preset number of iterations, and whether i is greater than or equal to N.

[0171] If so, execute S515.

[0172] If not, execute S513.

[0173] N can be the preset maximum number of iterations.

[0174] S513. Update the model parameters of the i-th intermediate model according to the loss value to obtain the (i+1)-th intermediate model.

[0175] For example, suppose the preset number of iterations is 5. When obtaining the loss value of the 8th intermediate model, it is determined that the loss value has been changing within the last 5 iterations. At this point, the model parameters of the 8th intermediate model are updated based on the loss value of the 8th intermediate model to obtain the 9th intermediate model.

[0176] S514, Update i to i+1.

[0177] After S514, execute S509.

[0178] S515. Determine the current i-th intermediate model as the initial model.

[0179] For example, suppose the preset number of iterations is 5, and the preset maximum number of iterations N is 15. When obtaining the loss value of the 15th intermediate model, it is determined that the loss value has not changed in the last 5 iterations. At this point, the initial model corresponding to the current model parameters, the 15th intermediate model, is determined as the target model.

[0180] S516. Train the initial model using the validation dataset to obtain the target model.

[0181] The validation dataset is used to test and fine-tune the model parameters of the initial model. Training the initial model using the validation dataset improves the accuracy of the target model's output predictions.

[0182] The model training method provided in this application involves obtaining a first dataset and training a model using the first dataset to obtain a residual network. The model parameters of the residual network are used to determine the initialization parameters of the first model. A second dataset is obtained and divided into a training dataset and a validation dataset. The initialized first model is trained iteratively using the training dataset, with image enhancement processing performed on the training dataset after each iteration. The loss value is updated based on the prediction results of the first model, and the convergence of the model is determined based on the loss value. When the loss value no longer changes within the most recent preset number of iterations, the initial model corresponding to the current model parameters is determined as the target model. In the above process, because multiple training iterations are performed, and the sample image features used in each training iteration are different, the target model can identify whether objects in images in various scenarios are using mobile phones, improving the accuracy of determining whether objects are using mobile phones through the target model.

[0183] Based on any of the above embodiments, the following, in conjunction with Figure 7 The process of using the model will be explained.

[0184] Figure 7 This is a flowchart illustrating a method for determining whether an object exhibits a preset behavior, as provided in an embodiment of this application. Please refer to... Figure 7 The method may include:

[0185] S701, Obtain the first video to be processed.

[0186] The first video to be processed can be obtained by real-time uploading video from surveillance cameras in factories, workshops, or warehouses.

[0187] S702, Determine the image frame in the first video.

[0188] In the first video, an image frame is determined every preset number of frames. The preset number of frames can range from 25 to 30.

[0189] For any given image frame, perform preprocessing operations on the corresponding image. Preprocessing operations can include any of the following: image denoising, contrast enhancement, brightness adjustment, and saturation adjustment.

[0190] S703. The image frame is processed by the target model to obtain the prediction result.

[0191] The prediction result includes a first image type, or the prediction result includes a second image type and the predicted position of the image region corresponding to the preset behavior in the image frame. The first image type is used to indicate that the object in the image frame does not have the preset behavior, and the second image type is used to indicate that the object in the image frame has the preset behavior.

[0192] For example, suppose we need to determine whether an object is using a mobile phone using a target model. The first image type is 0, and the second image type is 1. The image frames to be processed are image A and image B. The prediction results obtained by processing image A and image B using the target model are shown in Table 2.

[0193] Table 2

[0194] Prediction results Image type Predicted location Image A 0 — Image B 1 (x1, y1, w1, h1)

[0195] Based on Table 2, it can be determined that the prediction result for image A includes the first image type, indicating that the object in image A does not exhibit mobile phone usage behavior. The prediction result for image B includes the second image type, and the predicted location (x1, y1, w1, h1) of the image region corresponding to mobile phone usage behavior in image B is also determined. This indicates that the object in image B exhibits mobile phone usage behavior, and the location corresponding to this behavior is marked.

[0196] The method for determining whether an object exhibits a preset behavior provided in this application involves acquiring a first video and determining an image frame to be processed based on the first video. The image frame is then processed using a target model to obtain a prediction result, thereby determining whether an object in the image frame exhibits a preset behavior. In this process, the target model can determine the image's labeling result based on the image and its corresponding image features. The target model can extract features from the entire image, not just hand contour features, and can determine whether an object in the image is using a mobile phone in different scenarios based on the labeling result, thus improving the accuracy of determining whether mobile phone use is occurring.

[0197] Figure 8 This is a schematic diagram of a model training device provided in an embodiment of this application. The model training device can be a chip or a chip module. Please refer to... Figure 8 The model training device 10 may include:

[0198] The first acquisition module 11 is used to acquire a first data set, which includes multiple first sample images and image features of each first sample image;

[0199] The first training module 12 is used to train the model using the first dataset to obtain a residual network. The residual network includes at least one convolutional block and at least one residual block, and each convolutional block and the residual block is followed by an average pooling layer and a convolutional layer.

[0200] The second acquisition module 13 is used to acquire a second data set, which includes multiple second sample images and a labeling result for each second sample image. The labeling result is used to indicate whether an object in the second sample image has a preset behavior.

[0201] The second training module 14 is used to train the model using the second dataset to obtain a target model. The target model includes the residual network and the multi-feature fusion network. The multi-feature fusion network includes a spatial pyramid pooling layer and a regularized convolutional layer. The target model is used to identify whether an object in an image exhibits the preset behavior.

[0202] The model training device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0203] In one possible implementation, the second training module 14 is specifically used for:

[0204] In the second dataset, determine the training dataset and the validation dataset;

[0205] The model is trained using the training dataset to obtain an initial model;

[0206] The initial model is trained using the validation dataset to obtain the target model.

[0207] In one possible implementation, the second training module 14 is specifically used for:

[0208] The model parameters of the first model are initialized using the model parameters of the residual network to obtain the initialized first model;

[0209] The training dataset is subjected to multiple image enhancement processes to obtain multiple enhanced datasets;

[0210] The initial model is obtained by iteratively training the first model after initialization using the training dataset and the multiple augmentation datasets respectively.

[0211] In one possible implementation, the second training module 14 is specifically used for:

[0212] The first initialized model is trained in the first iteration using the training dataset to obtain the first intermediate model;

[0213] The i-th intermediate model is trained for the (i+1)th iteration using the i-th augmented dataset to obtain the (i+1)-th intermediate model. The i-th intermediate model is obtained by taking the values ​​1, 2, 3, ..., until the i-th intermediate model converges. When i is greater than or equal to N, the i-th intermediate model is determined as the initial model, where N is an integer greater than 1.

[0214] In one possible implementation, the second training module 14 is specifically used for:

[0215] The i-th intermediate model processes the sample images in the i-th augmented dataset to obtain prediction results, which include the predicted positions of the prediction boxes.

[0216] The marked area of ​​the marked box is determined based on the marked position, and the predicted area of ​​the predicted box is determined based on the predicted position;

[0217] The loss value is determined based on the predicted location, the predicted area, the marked location, the marked area, and a preset loss function;

[0218] The model parameters of the i-th intermediate model are updated based on the loss value to obtain the (i+1)-th intermediate model.

[0219] In one possible implementation, the second training module 14 is specifically used for:

[0220] The third sample image is obtained by performing image enhancement processing on the second sample image in the training dataset. The image enhancement processing includes at least one of the following: scaling, flipping, translation, rotation, grayscale filling, mean normalization, variance standardization, color transformation, image blending, and random multi-scale operation.

[0221] For any second sample image, the labeling result corresponding to the second sample image is adaptively adjusted to obtain the labeling result corresponding to the third sample image.

[0222] In one possible implementation, the second training module 14 is specifically used for:

[0223] The first labeling result includes a first image type, which indicates that the object in the second sample image does not exhibit the preset behavior; or,

[0224] The second labeling result includes a second image type and the labeling position of the image region corresponding to the preset behavior in the second sample image. The second image type is used to indicate that the object in the second sample image has the preset behavior.

[0225] In one possible implementation, the second training module 14 is specifically used for:

[0226] If the labeling result corresponding to the second sample image is the first labeling result, then the labeling result corresponding to the third sample image is determined to be the first labeling result;

[0227] If the labeling result corresponding to the second sample image is the second labeling result, then the labeling position in the second labeling result is updated according to the image enhancement process to obtain the updated labeling position, and the labeling result corresponding to the third sample image is determined to include the updated labeling position and the second image type.

[0228] The model training device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0229] Figure 9 This is a schematic diagram of another model training device provided in an embodiment of this application. Figure 8 Based on the illustrated embodiments, please refer to Figure 9 The model training device 10 also includes a third acquisition module 15, a determination module 16, and a processing module 17.

[0230] The third acquisition module 15 is used for:

[0231] Get the first video to be processed.

[0232] The determining module 16 is used for:

[0233] Identify image frames in the first video.

[0234] The processing module 17 is used for:

[0235] The image frame is processed by the target model to obtain the prediction result.

[0236] The prediction result includes a first image type, or the prediction result includes a second image type and the predicted position of the image region corresponding to the preset behavior in the image frame. The first image type is used to indicate that the object in the image frame does not have the preset behavior, and the second image type is used to indicate that the object in the image frame has the preset behavior.

[0237] The model training device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0238] Figure 10 This is a schematic diagram of the model training device provided in an embodiment of this application. Please refer to... Figure 10 The model training device 20 may include a memory 21 and a processor 22. Exemplarily, the memory 21 and the processor 22 are interconnected via a bus 23.

[0239] Memory 21 is used to store program instructions;

[0240] The processor 22 is used to execute the program instructions stored in the memory, so that the model training device 20 performs the method shown in the above method embodiment.

[0241] The model training device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0242] This application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above-described method when executed by a processor.

[0243] This application embodiment may also provide a computer program product, including a computer program that, when executed by a processor, can implement the above-described method.

[0244] All or part of the steps in the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above-described method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), random access memory (RAM), flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0245] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0246] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0247] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0248] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0249] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

Claims

1. A model training method, characterized in that, include: Obtain a first data set, which includes multiple first sample images and image features of each first sample image; The model is trained using a first dataset to obtain a residual network, which includes at least one convolutional block and at least one residual block, with an average pooling layer and a convolutional layer inserted after each convolutional block and the residual block. Obtain a second data set, which includes multiple second sample images and the labeling results for each second sample image. The labeling results include the labeling positions of the bounding boxes. The labeling results are used to indicate whether the objects in the second sample images exhibit a preset behavior. The preset behavior is the behavior of using a mobile phone; The target model is obtained by training the model using a second dataset. The target model includes the residual network and a multi-feature fusion network. The multi-feature fusion network includes a spatial pyramid pooling layer and a regularized convolutional layer. The spatial pyramid pooling layer is used to fuse the features extracted by the residual network under different receptive fields. The regularized convolutional layer is used to randomly delete some feature information so that the target model can randomly learn the feature information that has not been deleted multiple times. The target model is used to identify whether the object in the image has the preset behavior. The target model is obtained by training the model using the second dataset, including: In the second dataset, determine the training dataset and the validation dataset; The model is trained using the training dataset to obtain an initial model; The initial model is trained using the validation dataset to obtain the target model; The initial model is obtained by training the model using the training dataset, including: The model parameters of the first model are initialized using the model parameters of the residual network to obtain the initialized first model; The training dataset is subjected to multiple image enhancement processes to obtain multiple enhanced datasets; The first initialized model is trained in the first iteration using the training dataset to obtain the first intermediate model; The i-th intermediate model processes the sample images in the i-th augmented dataset to obtain prediction results, which include the predicted positions of the prediction boxes. The marked area of ​​the marked box is determined based on the marked position, and the predicted area of ​​the predicted box is determined based on the predicted position; The loss value is determined based on the predicted location, the predicted area, the marked location, the marked area, and a preset loss function; The model parameters of the i-th intermediate model are updated according to the loss value to obtain the (i+1)-th intermediate model. The i-th intermediate model is obtained by taking the values ​​1, 2, 3, ..., until the i-th intermediate model converges. When i is greater than or equal to N, the i-th intermediate model is determined as the initial model. N is an integer greater than 1.

2. The method according to claim 1, characterized in that, This refers to any one of the multiple image enhancement processes; The training dataset is subjected to image augmentation processing to obtain an augmented dataset, including: The third sample image is obtained by performing image enhancement processing on the second sample image in the training dataset. The image enhancement processing includes at least one of the following: scaling, flipping, translation, rotation, grayscale filling, mean normalization, variance standardization, color transformation, image blending, and random multi-scale operation. For any second sample image, the labeling result corresponding to the second sample image is adaptively adjusted to obtain the labeling result corresponding to the third sample image.

3. The method according to claim 2, characterized in that, The labeling results include a first labeling result and a second labeling result, wherein, The first labeling result includes a first image type, which indicates that the object in the second sample image does not exhibit the preset behavior; or, The second labeling result includes a second image type and the labeling position of the image region corresponding to the preset behavior in the second sample image. The second image type is used to indicate that the object in the second sample image has the preset behavior.

4. The method according to claim 3, characterized in that, Adaptive adjustment is performed on the labeling results corresponding to the second sample image to obtain the labeling results corresponding to the third sample image, including: If the labeling result corresponding to the second sample image is the first labeling result, then the labeling result corresponding to the third sample image is determined to be the first labeling result; If the labeling result corresponding to the second sample image is the second labeling result, then the labeling position in the second labeling result is updated according to the image enhancement process to obtain the updated labeling position, and the labeling result corresponding to the third sample image is determined to include the updated labeling position and the second image type.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Get the first video to be processed; Determine image frames in the first video; The image frame is processed using the target model to obtain the prediction result; The prediction result includes a first image type, or the prediction result includes a second image type and the predicted position of the image region corresponding to the preset behavior in the image frame. The first image type is used to indicate that the object in the image frame does not have the preset behavior, and the second image type is used to indicate that the object in the image frame has the preset behavior.

6. A model training device, characterized in that, The device includes: The first acquisition module is used to acquire a first data set, which includes multiple first sample images and image features of each first sample image; The first training module is used to train the model using a first dataset to obtain a residual network, wherein the residual network includes at least one convolutional block and at least one residual block, and an average pooling layer and a convolutional layer are inserted after each convolutional block and the residual block. The second acquisition module is used to acquire a second data set, which includes multiple second sample images and a labeling result for each second sample image. The labeling result includes the labeling position of the label box. The labeling result is used to indicate whether an object in the second sample image has a preset behavior. The preset behavior is the behavior of using a mobile phone. The second training module is used to train the model using the second dataset to obtain the target model. The target model includes the residual network and the multi-feature fusion network. The multi-feature fusion network includes a spatial pyramid pooling layer and a regularized convolutional layer. The spatial pyramid pooling layer is used to fuse the features extracted by the residual network under different receptive fields. The regularized convolutional layer is used to randomly delete some feature information so that the target model can randomly learn the feature information that has not been deleted multiple times. The target model is used to identify whether the object in the image has the preset behavior. When the target model is obtained by training the model using the second dataset, the second training module is specifically used to determine the training dataset and the validation dataset in the second dataset; train the model using the training dataset to obtain an initial model; and train the initial model using the validation dataset to obtain the target model. When training the model using the training dataset to obtain an initial model, the second training module is specifically used to initialize the model parameters of the first model using the model parameters of the residual network to obtain the initialized first model. The training dataset is subjected to multiple image enhancement processes to obtain multiple enhanced datasets; The first initialized model is trained in the first iteration using the training dataset to obtain the first intermediate model; The i-th intermediate model processes the sample images in the i-th augmented dataset to obtain prediction results, which include the predicted positions of the prediction boxes. The marked area of ​​the marked box is determined based on the marked position, and the predicted area of ​​the predicted box is determined based on the predicted position; The loss value is determined based on the predicted location, the predicted area, the marked location, the marked area, and a preset loss function; The model parameters of the i-th intermediate model are updated according to the loss value to obtain the (i+1)-th intermediate model. The i-th intermediate model is obtained by taking the values ​​1, 2, 3, ..., until the i-th intermediate model converges. When i is greater than or equal to N, the i-th intermediate model is determined as the initial model. N is an integer greater than 1.

7. A model training device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images