Model training method and device and storage medium

Abstract

The invention provides a model training method and device and a storage medium, relates to the technical field of computers, and can improve the recognition accuracy of an image detection model as much as possible. The method comprises: a computing device obtains a sample image of an object in a dark environment, and extracts initial feature information of the object in the sample image based on a first convolutional network, the first convolutional network being a convolutional network after activation value binarization; furthermore, under the condition that the similarity between the initial feature information extracted by the first convolutional network and the actual feature information of the object is smaller than a first preset threshold value, the computing device adjusts network parameters of the first convolutional network to obtain a second convolutional network; and the computing device trains the sample image of the object based on the second convolutional network to obtain an image detection model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, device and storage medium. Background Art

[0002] Object detection algorithms are an important research direction in the field of computer vision, which can detect and identify target objects in images or videos. Object detection is widely used in a variety of devices, such as vehicle dashcams, smart phones, smart monitoring devices, etc. In order to adapt to the different scene requirements of the recognition tasks that can be performed, the detection performance of object detection algorithms in dark environments has received attention and development.

[0003] When the computing resources and storage resources of the above-mentioned devices are limited, if a target detection algorithm is used, the parameters involved in the target detection algorithm need to be compressed. However, the existing parameter compression technology significantly reduces the accuracy of the parameters involved in the target detection algorithm, resulting in a significant decrease in recognition accuracy when performing target recognition tasks in a dark environment. Summary of the invention

[0004] The present application provides a model training method, device and storage medium, which can improve the recognition accuracy of the image detection model as much as possible.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a model training method, which includes: obtaining a sample image of an object in a dark light environment; based on a first convolutional network, extracting initial feature information of the object in the sample image; the first convolutional network is a convolutional network after activation value binarization; when the similarity between the initial feature information and the actual feature information of the object is less than a first preset threshold, adjusting the network parameters of the first convolutional network to obtain a second convolutional network; training the sample image based on the second convolutional network to obtain an image detection model.

[0007] In one possible implementation, the network parameters of the first convolutional network are adjusted to obtain the second convolutional network, including: based on the loss function corresponding to the similarity, performing gradient calculation on the weight value of the first convolutional network to obtain the gradient value; based on the gradient value, adjusting the network parameters of the first convolutional network to obtain the second convolutional network.

[0008] In one possible implementation, the sample image is trained based on the second convolutional network to obtain an image detection model, including: binarizing the weight values ​​of each convolutional layer in the second convolutional network to obtain a processed second convolutional network; and training the sample image based on the processed second convolutional network to obtain an image detection model.

[0009] In one possible implementation, after binarizing the weight values ​​of each convolutional layer in the second convolutional network to obtain the processed second convolutional network, the method further includes: adjusting the weight values ​​of each convolutional layer based on the scaling factor so that the similarity between the feature information extracted from the sample image by the second convolutional network and the actual feature information of the object is greater than a second preset threshold.

[0010] In a possible implementation, the method also includes: obtaining a video to be detected; the video to be detected is shot of the object to be detected; inputting the video to be detected into an image detection model respectively to obtain one or more detection results; the detection result includes a detection box corresponding to the object to be detected; and filtering one or more detection results based on a preset auxiliary screening strategy to obtain a target detection result.

[0011] In one possible implementation, the preset auxiliary screening strategy includes at least one of the following: a continuous frame auxiliary judgment strategy; the continuous frame judgment strategy refers to taking the detection results of N consecutive frames of images as the target detection results; the offset value between the detection frames of the objects to be detected in the detection results of N frames of images is less than a preset distance; N is a positive integer; a non-maximum suppression strategy; the non-maximum suppression strategy refers to determining the target detection results based on the intersection-and-union ratio and confidence of the detection frames of the objects to be detected in each frame of the image; a confidence filtering strategy; the confidence filtering strategy refers to filtering the detection results whose confidence is less than a preset threshold.

[0012] In a second aspect, the present application provides a model training device, which includes: a communication unit and a processing unit; the communication unit is used to obtain a sample image of an object in a dark light environment; the processing unit is used to extract initial feature information of the object in the sample image based on a first convolutional network; wherein the first convolutional network is a convolutional network after the activation value is binarized; the processing unit is also used to adjust the network parameters of the first convolutional network to obtain a second convolutional network when the similarity between the initial feature information and the actual feature information of the object is less than a first preset threshold; the processing unit is also used to train the sample image based on the second convolutional network to obtain an image detection model.

[0013] In one possible implementation, the processing unit is specifically used to perform gradient calculation on the weight value of the first convolutional network based on the loss function corresponding to the similarity to obtain the gradient value; the processing unit is specifically used to adjust the network parameters of the first convolutional network based on the gradient value to obtain the second convolutional network.

[0014] In one possible implementation, the processing unit is specifically used to binarize the weight values ​​of each convolutional layer in the second convolutional network to obtain a processed second convolutional network; the processing unit is specifically used to train the processed second convolutional network based on the sample image to obtain an image detection model.

[0015] In one possible implementation, after binarizing the weight values ​​of each convolutional layer in the second convolutional network to obtain the processed second convolutional network, the processing unit is further used to adjust the weight values ​​of each convolutional layer based on the scaling factor so that the similarity between the feature information extracted from the sample image by the second convolutional network and the actual feature information of the object is greater than a second preset threshold.

[0016] In one possible implementation, the communication unit is also used to obtain the video to be detected; the video to be detected is shot of the object to be detected; the processing unit is also used to input the video to be detected into the image detection model respectively to obtain one or more detection results; wherein the detection result includes a detection box corresponding to the object to be detected; the processing unit is also used to filter one or more detection results based on a preset auxiliary screening strategy to obtain a target detection result.

[0017] In one possible implementation, the preset auxiliary screening strategy includes at least one of the following: a continuous frame auxiliary judgment strategy; the continuous frame judgment strategy refers to taking the detection results of N consecutive frames of images as the target detection results; the offset value between the detection frames of the objects to be detected in the detection results of N frames of images is less than a preset distance; N is a positive integer; a non-maximum suppression strategy; the non-maximum suppression strategy refers to determining the target detection results based on the intersection-and-union ratio and confidence of the detection frames of the objects to be detected in each frame of the image; a confidence filtering strategy; the confidence filtering strategy refers to filtering the detection results whose confidence is less than a preset threshold.

[0018] In a third aspect, the present application provides a model training device, comprising: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement the model training method described in the first aspect and any possible implementation method of the first aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal, the terminal executes the model training method described in the first aspect and any possible implementation of the first aspect.

[0020] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when the computer program product is run on a model training device, enables the model training device to perform the model training method described in the first aspect and any possible implementation of the first aspect.

[0021] In a sixth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement the model training method described in the first aspect and any possible implementation method of the first aspect.

[0022] Specifically, the chip provided in the present application also includes a memory for storing computer programs or instructions.

[0023] In the model training method provided in the present application, after obtaining a sample image of an object in a dark light environment, the computing device can input the sample image of the object in the dark light environment into a first convolutional network with binarized activation values. Accordingly, the first convolutional network outputs initial feature information of the object, so that the computing device determines the similarity between the initial feature information of the object output by the first convolutional network and the actual feature information of the object based on the initial feature information of the object output by the first convolutional network and the actual feature information of the object.

[0024] When the similarity between the initial feature information of the object output by the first convolutional network and the actual feature information of the object is less than a first preset threshold, the computing device adjusts the parameters of the first convolutional network. Since the computing device binarizes the activation values ​​of the convolutional layer of the first convolutional network, the parameters in the first convolutional network adapt to the constraints of the binarized activation values ​​during the process of the computing device adjusting the parameters of the first convolutional network. The constraints imposed by the binarized activation values ​​on the adjustment of the parameters of the first convolutional network provide a basis for parameter update for further adjustment of the parameters of the subsequent second convolutional network, thereby improving the training efficiency of the subsequent computing device for the training of the second convolutional network.

[0025] Furthermore, the computing device trains the second convolutional network to obtain an image detection model. That is, during the training of the convolutional network, the computing device gradually adjusts the parameters of the convolutional layer in the convolutional network to obtain the final graphic detection model. Compared with using binary neural network technology to directly compress the full-precision parameters of the model to reduce the computing resources and storage resources of the model, the model training method provided in the present application improves the recognition accuracy of the image detection model as much as possible while compressing the parameter system of the image detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the structure of a model training system provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the composition of a model training device provided in an embodiment of the present application;

[0028] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0029] Figure 4 A flowchart of another model training method provided in an embodiment of the present application;

[0030] Figure 5 A flowchart of another model training method provided in an embodiment of the present application;

[0031] Figure 6 A flowchart of another model training method provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The model training method, device and storage medium provided in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0034] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0035] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0036] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0037] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0038] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0039] The target detection model is an important research direction in the field of computer vision. The target detection model detects the target object by extracting the feature information of the target object in the image or video, and identifies the category of the target object and the location of the target object in the image or video. The target detection model is widely used in a variety of devices, such as vehicle dashcams, smart phones, smart monitoring devices, etc. The above-mentioned devices realize the detection and identification of the target object by deploying the target detection model, and then further process the target object based on the relevant information of the target object identified by the target detection model. For example, a target detection model that can detect and identify cultural relics displayed in a museum is deployed in virtual reality (VR) glasses. VR glasses can present the relevant background knowledge of the corresponding cultural relics to tourists through the display screen of VR glasses in the form of text, pictures, or animations based on the information of the corresponding cultural relics identified by the target detection model. Accordingly, tourists can start the cultural relic viewing mode by wearing VR glasses during the viewing process to interact with the cultural relics displayed in the museum to enrich the viewing experience of tourists.

[0040] However, when the target detection model is used to perform the detection task of the target object in a dark light environment, due to the influence of the dark light environment, some or all of the features of the target object in the picture or video obtained by the target detection model will change color, darken, or even have unclear outlines. For example, in order to protect the exhibited cultural relics, the lights in the museum adopt a dark light mode. Therefore, when performing the recognition task of the target object in a dark light environment, the difference between the feature information of the target object extracted by the target detection model and the actual feature information of the target object may be large. In order to meet the needs of the recognition task in a dark light environment, the detection performance of the target detection model in a dark light environment has received attention and development, and the dark target detection model has gradually matured.

[0041] Some dark object detection models can increase the feature representation of the target object in the acquired image or video, that is, through a higher feature extraction capability of the target object, accurately detect and identify the target object in a dark environment. For example, the you only look once version (pyramid enhancement network for YOLO, PE-YOLO) dark object detection model is a dark object detection model that combines the pyramid enhancement network (PENet) with the YOLOv3 (YOLO version 3, YOLOv3) network.

[0042] Among them, PENet uses the Laplacian pyramid to decompose the image into multiple images of different resolutions to capture the feature details of the image at different scales. PENet includes a detail processing module (DPM) and a low-frequency enhancement filter (LEF). DPM is used to enhance image details through context branches and edge branches. Among them, the context branch is responsible for obtaining context information, and the edge branch obtains the edge information of the target by calculating the gradient of the image. LEF is used to capture low-frequency semantic information and reduce high-frequency noise. LEF intercepts low-frequency components through adaptive average pooling operations and allows information below the cutoff frequency to pass. Due to its high feature extraction capability, the recognition accuracy of the PE-YOLO dark target detection model on the dark data set is significantly higher than that of target detection models such as YOLOv3 and Zero-DCE.

[0043] In addition, when the target detection model is deployed on the device and applied in actual scenarios, the recognition efficiency of the dark target detection model is closely related to the user's experience, that is, the target detection model must have a fast enough recognition efficiency so that the device can display the content required by the user as timely as possible. At present, some target detection models have taken into account both high recognition efficiency and high recognition accuracy. For example, the spiking convolution detection neural network (SCOD). SCOD consists of two parts, of which the first part is a pre-trained 16th version of the convolutional neural network model (visual geometry group 16, VGG16), which uses stacked convolutional layers and fully connected layers, and performs pooling operations after each convolutional layer to extract image features and details. The second part is a combination of a spiking neural network layer and a normal convolutional layer, which is used to detect image bounding boxes and reduce the computational cost of the model. The model maintains a high accuracy recognition rate in the PASCAL Visual Object Classes Challenge 2012 (VOC12) dataset and the Ex-Dark dataset, and has high computational efficiency.

[0044] Since the equipment deployed with the dark target detection model has high requirements for the model's feature extraction capability, model generalization capability, and model recognition accuracy, the parameters of the dark target detection model are larger than those of the ordinary target detection model. Therefore, if the dark target detection model is to be deployed in a device with less computing resources and storage resources (for example, an end device), the dark target detection model needs to be quantized to compress the parameters in the dark target detection model. The binary neural network (BNN) technology is the quantization technology with the highest degree of compression of the parameters in the dark target detection model. By directly representing the complex parameters of some network layers in the target detection model with a bit 0 or 1, the parameters of the dark target detection model can be significantly compressed.

[0045] It should be noted that the binary neural network technology can compress the parameters of the dark target detection model through the symbolic function.

[0046] However, due to the excessive degree of compression of the parameters of the dark target detection model by the binary neural network technology, the parameter accuracy of the dark target detection model quantized by the binary neural network technology is significantly reduced. Therefore, compared with the target detection model under full-precision parameters, the parameter values ​​of the quantized target detection model will be distorted, which will lead to a significant decrease in the recognition accuracy and generalization ability of the quantized dark target detection model when performing recognition tasks in a dark light environment.

[0047] At present, the binary neural network bilinear optimization method can be used to minimize the impact of model quantization on the recognition accuracy and generalization ability of the dark target detection model. Specifically, the binary neural network bilinear optimization method refers to the process in which the computing device uses the binary neural network technology to compress the weight value of the convolution layer in the convolution network of the dark target detection model into a binary weight, and introduces a scaling factor into the convolution layer of the dark target detection model, so that the product of the scaling factor and the compressed binary weight of the convolution layer is as close as possible to the weight value of the convolution layer. In the process of training the dark target detection model based on the binary neural network bilinear optimization method, the computing device uses the product of the scaling factor and the binary weight as the training weight of the convolution layer.

[0048] Further, the computing device inputs the image corresponding to the target object into the convolution layer of the dark target detection model, and accordingly, the convolution layer outputs feature information (also referred to as a feature map) of the target object. The computing device iteratively re-inputs the feature information of the object output by the convolution layer into the convolution layer to iteratively update the scaling factor and the weight value of the convolution layer.

[0049] Optionally, the binary neural network bilinear method satisfies the following formulas 1 to 3.

[0050]

[0051] Among them, w is the weight in the convolution layer, and each element in w represents a weight value of the convolution layer, w = (w1, w2, ... w n ) T α is the scaling factor, α=diag(α1,α2,…α i ). b w represents the binary weight corresponding to the weight value w, represents element-wise multiplication, R(w) is a regularization term, which represents the constraint relationship about the weight value w. Represents the bilinear form of the scaling factor α and the weight value w. express The square of the Euclidean distance between and w. Formula 1 indicates that the binary neural network bilinear method aims to find a set (w, α) that minimizes G(w, α). i is a positive integer, n is a positive integer, i represents the number of scaling factors, and n represents the number of weight values ​​of the convolutional layer.

[0052] In one example, if That is, A represents the inverse matrix of the scaling factor α, in which each element of A is the reciprocal of each scaling factor. Formula 1 can also be expressed using the following formula 2:

[0053]

[0054] The update of the weight value and scaling factor of the convolutional layer by the binary neural network bilinear optimization method satisfies the following formula 3.

[0055]

[0056] Where L is the loss function, which is used to represent the difference between the feature information of the object output by the convolution layer during training and the actual feature information of the object. t is the tth iteration, which indicates the tth update of the weight value and scaling factor of the convolution layer. in is the feature information of the input convolution layer, a out is the feature information output by the convolutional layer, c in Indicates the number of input channels, W in ×H in Represents the width and height of the feature information, and accordingly, Represents the gradient of the loss function L with respect to A at the tth iteration. express The gradient of A at iteration t. represents the gradient of the loss function L with respect to the feature information output by the convolutional layer at the tth iteration, Represents the gradient of the feature information output by the convolutional layer at the tth iteration with respect to the scaling factor updated for the tth time. Represents the binary weight corresponding to the feature information of the input convolutional layer.

[0057] However, the bilinear optimization method of binary neural network ignores the intrinsic bilinear relationship between the two variables of scaling factor and weight value. In the process of training the dark target detection model, only one of the variables is iteratively updated while the other variable is kept unchanged, resulting in the trained dark target detection model being an under-trained suboptimal model.

[0058] In addition, when dealing with out-of-domain generalization tasks, due to the bilinear constraint relationship between the scaling factor and the weight value, the scaling factor and the weight value will converge asynchronously during the training process of the dark target detection model. The asynchronous convergence of the scaling factor and the weight value of the convolutional layer will directly affect the learning process of the weight of the convolutional layer of the dark target detection model. Moreover, it is difficult to find the flat minimum value of the weight value generated in multiple iterations using the traditional random weight dense averaging method for the dark target detection model quantized by the binary neural network bilinear optimization method, resulting in poor recognition results for the target objects generalized out of the recognition domain of the dark target detection model.

[0059] Furthermore, the binary neural network bilinear optimization method will also be disturbed by the changes in feature distribution caused by the out-of-domain generalization tasks. In other words, the binary neural network bilinear optimization method cannot obtain a generalizable binary neural network technology, and the generalization ability of the quantized dark target detection model finally trained is insufficient.

[0060] In view of this, an embodiment of the present application provides a model training method, after the computing device obtains a sample image of an object in a dark light environment, the sample image of the object in the dark light environment can be input into a first convolutional network with a binary activation value. Accordingly, the first convolutional network outputs the initial feature information of the object, so that the computing device determines the similarity between the initial feature information of the object output by the first convolutional network and the actual feature information of the object based on the initial feature information of the object output by the first convolutional network and the actual feature information of the object.

[0061] When the similarity between the initial feature information of the object output by the first convolutional network and the actual feature information of the object is less than the first preset threshold, the computing device adjusts the network parameters of the first convolutional network. Since the computing device performs binarization processing on the activation values ​​of the convolutional layer of the first convolutional network, in the process of adjusting the parameters, the parameters in the first convolutional network adapt to the constraints of the binarized activation values, providing a parameter update basis for further adjustment of the parameters of the subsequent second convolutional network, thereby improving the training efficiency of the subsequent computing device for the second convolutional network.

[0062] Furthermore, the computing device trains the second convolutional network to obtain an image detection model. That is, during the training of the convolutional network, the computing device gradually adjusts the parameters of the convolutional layer in the convolutional network to obtain the final graphic detection model. Compared with using binary neural network technology to directly compress the full-precision parameters of the model to reduce the computing resources and storage resources of the model, the model training method provided in the present application improves the recognition accuracy of the image detection model as much as possible while compressing the parameter system of the image detection model.

[0063] For example, Figure 1 A schematic diagram of the structure of a model training system 10 provided in an embodiment of the present application is shown in FIG. The model training system 10 may include at least one computing device 101 and at least one terminal device 102, and the computing device 101 may be connected to the terminal device 102 for communication. Figure 1 Only one computing device 101 and one terminal device 102 are shown. The embodiment of the present application does not impose any restriction on the number of one computing device 101 and one terminal device 102.

[0064] In one possible implementation, the terminal device 102 is used to capture an image or video including one or more objects, wherein each of the one or more objects is displayed separately in the image or video; and send the image or video including the one or more objects to the computing device 101.

[0065] The computing device 101 is used to obtain a sample image of an object in a dark light environment based on an image or video including one or more objects in a dark light environment collected from the terminal device 102; based on a first convolutional network, extract initial feature information of the object in the sample image; the first convolutional network is a convolutional network after the activation value is binarized; when the similarity between the initial feature information of the object and the actual feature information of the object is less than a first preset threshold, adjust the network parameters of the first convolutional network to obtain a second convolutional network; train the second convolutional network based on the sample image of the object to obtain an image detection model.

[0066] Optionally, the computing device 101 may be a large-capacity device with wireless communication capabilities (eg, a server integrating a large number of boards), which may be deployed on land, including indoors or outdoors or in a vehicle.

[0067] Optionally, the terminal device may be a device with a wireless transceiver function, and this application does not impose any restrictions on this. For example, the terminal device may be a mobile terminal device (such as a mobile phone, a tablet computer, or VR glasses, etc.).

[0068] It should be noted that Figure 1 This is just an exemplary framework diagram. Figure 1 The number of nodes included in the Figure 1 In addition to the functional nodes shown, the model training system 10 may also include other nodes, and this application does not impose any restrictions on this.

[0069] The application scenarios of the embodiments of the present application are not limited. The system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It is known to those skilled in the art that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0070] When implementing it specifically, Figure 1 The equipment in Figure 2 The structure shown, or including Figure 2 Parts shown. Figure 2The present invention provides a schematic diagram of the composition of a model training device 20 provided in an embodiment of the present invention. The model training device 20 may be a computing device 101 or a chip or a system on a chip in the computing device 101. Alternatively, the model training device 20 may be an end device 102 or a chip or a system on a chip in the end device 102. Figure 2 As shown, the model training device 20 may include a processor 201 and a communication line 202.

[0071] Furthermore, the model training device 20 may also include a communication interface 203 and a memory 204. The processor 201, the memory 204 and the communication interface 203 may be connected via a communication line 202.

[0072] The processor 201 is a central processing unit (CPU), a general processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 201 may also be other devices with processing functions, such as circuits, devices, or software modules, without limitation.

[0073] The communication line 202 is used to transmit information between the components included in the model training device 20.

[0074] The communication interface 203 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 203 may be a module, a circuit, a communication interface or any device capable of achieving communication.

[0075] The memory 204 is used to store instructions, where the instructions may be computer programs.

[0076] The memory 204 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, etc., without limitation.

[0077] It should be noted that the memory 204 can exist independently of the processor 201, or can be integrated with the processor 201. The memory 204 can be used to store instructions or program codes or some data, etc. The memory 204 can be located in the model training device 20, or can be located outside the model training device 20, without limitation. The processor 201 is used to execute the instructions stored in the memory 204 to implement the model training method provided in the following embodiments of the present application.

[0078] In an example, the processor 201 may include one or more CPUs, for example, CPU0 and CPU1 (not shown in the figure).

[0079] As an optional implementation, the model training device 20 includes multiple processors.

[0080] As an optional implementation, the model training device 20 further includes an output device and an input device. Exemplarily, the input device is a keyboard, a mouse, a microphone, a joystick, and the like, and the output device is a display screen, a speaker, and the like.

[0081] It should be noted that the model training device 20 can be a desktop computer, a portable computer, a network server, a mobile phone, a tablet computer, a wireless terminal, an embedded device, a chip system or a Figure 2 In addition, Figure 2 The composition shown in the Figure 1 as well as Figure 2 The limitations of each device in Figure 2 In addition to the parts shown, Figure 1 as well as Figure 2 The various devices in the drawings may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0082] In the embodiment of the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices.

[0083] In addition, the actions, terms, etc. involved in the various embodiments of the present application can refer to each other without limitation. The message name or parameter name in the message exchanged between the various devices in the embodiments of the present application is only an example, and other names can also be used in the specific implementation without limitation.

[0084] The model training method provided by the embodiment of the present application is described below in conjunction with the accompanying drawings. Among them, the actions, terms, etc. involved in the various embodiments of the present application can refer to each other without limitation. The message name or parameter name in the message exchanged between the various devices in the embodiment of the present application is only an example, and other names can also be used in the specific implementation without limitation. The actions involved in the various embodiments of the present application are only an example, and other names can also be used in the specific implementation, such as: "included in" in the embodiment of the present application can also be replaced by "carried on" or "carried in", etc.

[0085] In order to solve the problems existing in the above-mentioned prior art, the embodiment of the present application proposes a model training method, which can improve the recognition accuracy of the image detection model as much as possible. Figure 3 As shown, the method includes:

[0086] S301. A computing device obtains a sample image of an object in a dark environment.

[0087] As a possible implementation, the implementation process of the above S301 may be: the terminal device collects an image or video including one or more objects in a dark environment, and sends the collected image or video to the computing device. The computing device may use the received image as a sample image, or extract a sample image from the received video.

[0088] In one scenario, when a computing device receives a video including one or more objects in a dark environment captured by an end device, the computing device divides the video including multiple objects in the dark environment into image data sets according to a preset frame rate, and determines the category to which the object in each image in the image data set belongs. The computing device classifies the image data sets according to the category to which the object belongs, and obtains multiple categories of image data sets. The computing device uses an image labeling tool to add a label set of the corresponding object to each image in the multiple categories of image data sets, and obtains sample images of multiple objects in the dark environment.

[0089] The image marking tool can add a detection box and a label set to the image, wherein the detection box surrounds all or part of the area of ​​the object in the image, and the label set includes the category information and location information of the object, etc. Specifically, the image marking tool in the prior art can be used without limitation.

[0090] Furthermore, the computing device may randomly divide the sample images of the object into a training set, a validation set, and a test set according to a preset ratio to ensure that the features in the above three sets corresponding to the sample images of the object are identically distributed but independent of each other.

[0091] In the present application, the above-mentioned preset frame rate and preset ratio can be set according to specific needs. For example, the preset frame rate can be 24 frames per second, and the preset ratio can be 8:1:1, without limitation.

[0092] In one example, the above tag set may also include shape information of the object and size information of the object, without limitation.

[0093] S302: The computing device extracts initial feature information of the object in the sample image based on the first convolutional network.

[0094] Among them, the first convolutional network is a convolutional network after the activation value is binarized.

[0095] As a possible implementation, the implementation process of S302 may be as follows: the computing device constructs a backbone network structure of the dark target detection model, and binarizes the activation values ​​of the convolution layer of the convolution network of the dark target detection model to obtain a first convolution network. The computing device inputs a training set (hereinafter referred to as the training set) of sample images of multiple objects into the first convolution network. Accordingly, the first convolution network may output initial feature information of the objects in the training set.

[0096] Among them, the computing device can binarize the activation value of the convolutional network through a symbolic function.

[0097] In one example, the dark target detection model may be a YOLOv10 model, or any other dark target detection model without limitation.

[0098] It should be noted that the backbone network of the dark target detection model may include convolutional layers, bottleneck layers, super performance processing framework (SPPF), and normalization layers. Among them, the YOLOv10 model can maintain optimal performance with fewer parameters, and there is a continuous dual allocation method that does not rely on non-maximum suppression training and an overall efficiency-accuracy driven model design strategy to achieve object recognition.

[0099] It is understandable that since the computing device trains the model on the backbone network structure of the dark object detection model, the dark object detection model can perform the object recognition task in a dark light environment. Therefore, the image detection model obtained by the computing device can perform the object recognition task in a dark light environment.

[0100] S303: When the similarity is less than a first preset threshold, the computing device adjusts the network parameters of the first convolutional network to obtain a second convolutional network.

[0101] As a possible implementation, the implementation process of S303 may be as follows: the computing device determines the similarity between the initial feature information output by the first convolutional network and the actual feature information of the object in the training set, and compares the determined similarity with the first preset threshold. When the similarity is less than the first preset threshold, the computing device trains the first convolutional network based on the training set, and adjusts the weight value of the convolutional layer in the first convolutional network during the training process to obtain the second convolutional network.

[0102] Among them, adjusting the weight value of the convolutional layer in the first convolutional network refers to the computing device determining the loss function based on the feature information output by the first convolutional network and the actual feature information of the object, and using the gradient of the determined loss function with respect to the weight value of the convolutional layer to update the weight value of the convolutional layer in the first convolutional network in back propagation. Back propagation refers to a method for calculating the gradient of the loss function of the weight value of the convolutional layer in the first convolutional network. The calculated gradient is used to indicate how to adjust the weight value of the convolutional layer in the first convolutional network to minimize the loss function.

[0103] It should be noted that the computing device can use the cosine similarity method to determine the similarity between the initial feature information output by the convolution layer and the actual feature information of the objects in the training set, or can use the Euclidean distance method to determine the similarity between the initial feature information output by the convolution layer and the actual feature information of the objects in the training set, without limitation.

[0104] In the present application, the first preset threshold can be set according to specific needs. For example, the first preset threshold can be 0.5, etc., without limitation.

[0105] S304: The computing device trains the sample image based on the second convolutional network to obtain an image detection model.

[0106] As a possible implementation method, the implementation process of the above S304 can be: in the process of training the training set based on the second convolutional network, the computing device uses the joint recursive optimization binarization method to iteratively update the weight value of the convolutional layer of the second convolutional network and obtain the image detection model.

[0107] In one scenario, before training the training set based on the second convolutional network, the computing device further compresses the parameters of the convolutional layer of the second convolutional network to obtain a compressed second convolutional network. The computing device trains the compressed second convolutional network based on the training set.

[0108] Further compressing the parameters means that the computing device performs binarization processing on the weight values ​​of the convolution layer of the second convolution network. The weights of the convolution layer of the compressed second convolution network are binary weights.

[0109] In the model training method provided in the present application, after obtaining a sample image of an object in a dark light environment, the computing device can input the sample image of the object in the dark light environment into a first convolutional network with binarized activation values. Accordingly, the first convolutional network outputs initial feature information of the object, so that the computing device determines the similarity between the initial feature information output by the first convolutional network and the actual feature information of the object based on the initial feature information of the object output by the first convolutional network and the actual feature information of the object.

[0110] When the similarity between the initial feature information of the object output by the first convolutional network and the actual feature information of the object is less than a first preset threshold, the computing device adjusts the parameters of the first convolutional network. Since the computing device binarizes the activation values ​​of the convolutional layer of the first convolutional network, the parameters in the first convolutional network adapt to the constraints of the binarized activation values ​​during the process of the computing device adjusting the parameters of the first convolutional network. The constraints imposed by the binarized activation values ​​on the adjustment of the parameters of the first convolutional network provide a basis for parameter update for further adjustment of the parameters of the subsequent second convolutional network, thereby improving the training efficiency of the subsequent computing device for the training of the second convolutional network.

[0111] Furthermore, the computing device trains the second convolutional network to obtain an image detection model. That is, during the training of the convolutional network, the computing device gradually compresses and adjusts the parameters of the convolutional layer in the convolutional network to obtain the final graphic detection model. Compared with using binary neural network technology to directly compress the full-precision parameters of the model to reduce the computing resources and storage resources of the model, the model training method provided in the present application improves the recognition accuracy of the image detection model as much as possible while compressing the parameter system of the image detection model.

[0112] As can be seen from the aforementioned description of "S303", the computing device adjusts the parameters of the convolutional layer of the first convolutional network based on the similarity between the initial feature information of the objects in the training set and the actual feature information of the objects in the training set to obtain the second convolutional network.

[0113] That is, the computing device may determine the loss function based on the similarity between the initial feature information of the object in the training set and the actual feature information of the object in the training set, wherein the loss function may express the difference between the initial feature information of the object in the training set and the actual feature information of the object in the training set. The training device adjusts the weight value of the convolution layer of the first convolution network based on the influence of the weight value of the first convolution network on the loss function to reduce the loss function. In view of this, if Figure 4 As shown, the above S303 can be specifically determined through the following S401 to S402.

[0114] S401. The computing device performs gradient calculation on the weight value of each convolutional layer in the first convolutional network based on the loss function corresponding to the similarity to obtain a gradient value.

[0115] Among them, the gradient calculation can determine the rate of change of the loss function with respect to the weights of the convolutional layer in the first convolutional network. Specifically, the gradient calculation refers to finding the direction in which the loss function increases the fastest.

[0116] In one example, the loss function may be a mean square error function of the initial feature information of the objects in the training set output by the first convolutional network and the actual feature information of the objects, or the loss function may be a cross entropy loss function of the initial feature information of the objects in the training set output by the first convolutional network and the actual feature information of the objects, without limitation.

[0117] S402. The computing device adjusts the network parameters of the first convolutional network based on the gradient value to obtain a second convolutional network.

[0118] As a possible implementation method, the implementation process of S402 above may be: the computing device adjusts the weight value of the first convolutional network in the back propagation based on the gradient value to obtain the adjusted first convolutional network. The computing device re-inputs the training set into the adjusted first convolutional network, and determines a new loss function based on the feature information of the objects in the training set output by the adjusted first convolutional network and the actual feature information of the objects in the training set. The computing device adjusts the weight value of the convolutional layer in the adjusted first convolutional network again in the back propagation based on the new loss function until the loss function is minimized, that is, the computing device iteratively adjusts the weight value of the convolutional layer in the first convolutional network until the loss function is minimized.

[0119] Among them, the specific implementation of the computing device iteratively adjusting the weight value of the convolutional layer in the first convolutional network based on the first gradient value in back propagation can refer to the existing technology and will not be described in detail here.

[0120] As can be seen from the above description of "S304", the computing device needs to train the second convolutional network to obtain the image detection model. That is to say, before obtaining the image detection model, the computing device also needs to process the weight values ​​of the convolutional layer in the second convolutional network. Then the computing device trains the second convolutional network to obtain the image detection model. In view of this, Figure 5 As shown, the above S304 can be specifically determined through the following S501 to S502.

[0121] S501. The computing device performs binarization processing on the weight values ​​of the convolutional layer in the second convolutional network to obtain a processed second convolutional network.

[0122] As a possible implementation, the implementation process of S501 above may be: the computing device determines a threshold. When the weight value of the convolutional layer of the second convolutional network is greater than or equal to the threshold, the computing device adjusts the weight value to 1; when the weight value of the convolutional layer of the second convolutional network is less than the threshold, the computing device adjusts the weight value to -1. In other words, the computing device adjusts the weight value of the convolutional layer in the second convolutional network to a binary weight to compress the parameters of the second convolutional network.

[0123] It should be noted that the above threshold can be set according to the range of the weight value of the convolutional layer of the second convolutional network, and the threshold can be any real number without restriction.

[0124] In one example, if the threshold is 0, the implementation process of S501 satisfies the following formula 4:

[0125]

[0126] Among them, sgn binarized (x) represents the binary symbol function, and x is the weight value of the convolutional layer of the second convolutional network.

[0127] It should be noted that the second convolutional network in the present application is obtained by adjusting the parameters of the first convolutional network by the computing device, that is, the activation values ​​in the second convolutional network are binary activation values. When the activation values ​​in the second convolutional network are binary activation values, the computing device adjusts the weight values ​​of the convolutional layer of the second convolutional network to binary weights. The present application compresses the parameters of the image detection model by binarizing the weight values ​​and activation values ​​of the convolutional layer in the convolutional network of the backbone network of the dark target detection model.

[0128] S502: The computing device trains the processed second convolutional network based on the scaling factor and the sample image of the object to obtain an image detection model.

[0129] The scaling factor can be used to retain the parameter accuracy of the processed second convolutional network. The scaling factor can be a random value or iteratively updated during the training of the second convolutional network.

[0130] In the present application, the difference between the product of the scaling factor and the binary weight of the second convolutional network and the weight value of the convolutional layer of the second convolutional network is less than a preset value. The preset value can be set according to the size of the specific weight value and the size of the scaling factor. For example, the preset value can be any real number without limitation.

[0131] As a possible implementation method, the implementation process of the above S502 can be: in the process of training the second convolutional network after the weight value is binarized based on the training set, the computing device introduces a scaling factor, and uses a joint recursive bilinear optimization method to iteratively update the scaling factor and the weight value of the convolutional layer of the second convolutional network to obtain an image detection model.

[0132] In addition, the similarity between the feature information of the object in the training set extracted by the second convolutional network and the actual feature information of the object is greater than a second preset threshold.

[0133] In the present application, the second preset threshold is greater than the first preset threshold, and the second preset threshold can be set according to specific needs. For example, the second preset threshold can be 0.8, etc., without limitation.

[0134] Optionally, in the above formula 3, if the weight value w approaches 0 due to the constraint of the regularization term R(w), The existence of will cause the magnitude of the reciprocal of the scaling factor A to increase. A too large magnitude may cause the gradient disappears, and the values ​​in the square of the inverse matrix of the scaling factor may become unstable.

[0135] Furthermore, in Formula 3 and In order to suppress the magnitude of the scaling factor A as much as possible, the joint recursive bilinear optimization method proposed in this application determines the update rule of the weight value of the convolutional layer of the new second convolutional network based on the bilinear relationship between the scaling factor and the weight value of the convolutional layer of the second convolutional network, and proposes a density function to supervise the update process of the weight of the second convolutional network.

[0136] Optionally, the above-mentioned joint recursive bilinear optimization method can satisfy the following formulas 5 to 14.

[0137]

[0138] Among them, the inverse matrix of the scaling factor The weights of the convolutional layer The weight of the convolution layer of the second convolutional network is a four-dimensional tensor, where K×K represents the size of the convolutional kernel when the height and width of the convolutional kernel in the second convolutional network are both K, that is, the convolutional kernel in the second convolutional network is a two-dimensional array of K×K. i，j Represents the weight value located in the i-th row and j-th column of the convolution kernel. represents the weight value of the weight value updated for the t+1th time considering the bilinear relationship, and η is the learning rate of the weight. i=1,2,…I,j=1,2,…J, where I and J are both positive real numbers.

[0139] In one example, if a weight value of the convolutional layer is Then 12 Represents the weight value of the first input channel, the second output channel, the first row, and the second column in the convolution kernel in the second convolutional network.

[0140] It can be understood that the above formula 5 shows the updating process of the weight value in the convolution layer under a fixed output channel and a fixed number of input channels. Formula 5 indicates that the updating rule of the weight value of the second convolutional network provided in the present application takes into account the bilinear relationship between the scaling factor and the weight value of the convolutional layer, and iteratively updates the scaling factor and the weight value of the convolutional layer in the second convolutional network to minimize the loss function as much as possible. Minimizing the loss function as much as possible can improve the feature extraction capability of the second convolutional layer, thereby improving the recognition accuracy of the image detection model.

[0141] Optionally, omit the The superscript of The following formula 6 is satisfied.

[0142]

[0143] It should be noted that when A ii The weight value w of the convolutional layer ij When finding partial derivatives, in A ii The number of rows and w ij If the number of rows is the same, If it exists, otherwise, the scaling factor is independent of the weight value of the convolutional layer, and the partial derivative of the scaling factor with respect to the convolutional layer is 0.

[0144] Furthermore, we can transform Expressed in vector form, And the weight of the i-th row in Formula 6 is expressed as a vector form,

[0145] Optionally, in combination with Formula 6, The following formula 7 is satisfied.

[0146]

[0147] Optionally, by combining Formula 6 and Formula 7, The following formula 8 is satisfied.

[0148]

[0149] As a possible implementation method, combined with formula 8, the above formula 5 The i-th component of satisfies the following formula 9.

[0150]

[0151] Optionally, in combination with Formula 5 to Formula 9, Formula 5 satisfies the following Formula 10.

[0152]

[0153] Alternatively, in Formula 10 The following formula 11 is satisfied.

[0154]

[0155] Optionally, Formula 11 represents the formula Defined as d t , d t It is unsolvable and undefined in the back-propagation of BNN. In order to determine d t The application uses a cyclic backtracking model to approximate d t , to loop back the weights w of the second convolutional network.

[0156] As a possible implementation method, the loop backtracking model satisfies the following formula 12.

[0157]

[0158] Wherein, U in Formula 12 represents the approximate learning rate. t ,A t ) is an activation function, which can introduce nonlinear relationships into the second convolutional network, thereby enhancing the learning ability of the second convolutional network. t ,A t ) is a supervision function provided by this application to supervise the update process of the cyclic backtracking model.

[0159] Furthermore, combined with Formula 12, DReLU(w t ,A t ) satisfies the following formula 13.

[0160]

[0161] in, w′ m is the mth component in w′, m=1,2,…C out . F(x) is the density function. In Denotes the value of D(w′ m ) is the inverse of the value of . For example, D(w′ m ) is 1, then is 0, D(w′ m ) is 0, then is 1.

[0162] It should be noted that the density function D(x) is expressed by Determine when asynchronous convergence occurs during the update, and implement the weights of the convolutional layer of the loop backtracking model Supervision of the convergence process of iterative updates.

[0163] Optionally, the density function D(x) is used to evaluate the density of the mth component of the variable in the variable system, and D(x) satisfies the following formula 14.

[0164]

[0165] Among them, σ(z) i Determine the prediction score of x based on the input variable z, ranking(σ(x) i ) represents σ(x) i The position or level in the set of σ(x) corresponding to all variables. T represents the density threshold. If σ(x) i If the position or level in the set of σ(x) corresponding to all variables exceeds T, the density function output is 1, otherwise, the density function output is 0.

[0166] In Formula 14, the value of T can be set according to specific needs. For example, the value of T can be any positive integer without limitation.

[0167] It can be understood that the computing device optimizes the weight value and scaling factor of the convolutional layer of the second convolutional network based on the joint recursive bilinear optimization method represented by Formulas 5 to 14 above, which can achieve sufficient training of the second convolutional network after weight compression and minimize the loss function as much as possible, thereby improving the recognition accuracy of the image detection model.

[0168] Furthermore, the joint recursive bilinear optimization method represented by the above formulas 5 to 14 defined in the embodiments recorded in this application reduces as much as possible the phenomenon of asynchronous convergence of the scaling factor and the weight value of the convolutional layer of the second convolutional network caused by the characteristics of non-convex optimization in the binary bilinear optimization method. Among them, this phenomenon of asynchronous convergence can cause the image detection model to have a local optimal problem. The problem of local optimality refers to the fact that when the image detection model detects an object in an image, it focuses on the local features or regions of the object, while ignoring the overall structure and contextual information of the object, which leads to incomplete, biased or false detection of the object.

[0169] In addition, the joint recursive bilinear optimization method described in this application can find the flat minimum of the loss function obtained multiple times during the training process of the second convolutional network, ensuring that the obtained image detection model can handle object detection tasks generalized outside the domain and has good generalization performance.

[0170] In addition, as some possible implementations, in the embodiments described in this application, the computing device may convert the format of the image detection model so that the image conversion model can be deployed in the terminal device. Before sending the converted image detection model to the terminal device, the computing device may verify and test the performance of the converted image detection model.

[0171] The following describes the process of "computing device verifying the performance of the image detection model".

[0172] As a possible implementation method, the image detection model obtained by the computing device is in .pth format. Since the terminal device usually cannot parse the image detection model in .pth format, the computing device needs to convert the image detection model in .pth format into an image detection model in .onnx format. Among them, the image detection model in .onnx format can be deployed across platforms, that is, the image detection model in .onnx format can be deployed in the terminal device.

[0173] Further, after obtaining the image detection model in .onnx format, the computing device converts the verification set (hereinafter referred to as the verification set) in the sample image of the above object into a tensor format to obtain a verification set in a tensor format. The verification set in a tensor format includes positive sample images and negative sample images corresponding to the above object.

[0174] The computing device inputs the validation set in tensor format into the image detection model. If the recognition results output by the image detection model for the positive sample images in the validation set in tensor format and the recognition results output by the image detection model for the negative sample images in the validation set are overfitted or underfitted, the computing device adjusts the hyperparameters involved in the training process of the second convolutional network to reduce the possibility of overfitting or underfitting recognition results. The computing device obtains the image detection model after the hyperparameters are adjusted.

[0175] In one example, the hyperparameters may include at least one of the following: initial learning rate, filtering threshold, intersection over union (IoU) threshold in non-maximum suppression (NMS) technology. Of course, the above is only an exemplary description of the hyperparameters, and the hyperparameters may also include other parameters, and this application does not impose any restrictions on this.

[0176] The following describes the process of "computing equipment testing the performance of the image detection model".

[0177] In the present application, after obtaining the image detection model after the hyperparameters are adjusted, the computing device tests the object recognition performance of the image detection model based on the test set of sample images of the above-mentioned object (hereinafter referred to as the test set). If the recognition results output by the image detection model for the test set have problems such as over-dependence on local features, misidentification or omission of object categories, and overlapping recognition frames, the computing device determines a new training set based on the sample images of the objects with inaccurate recognition results in the test set, and further trains on the new training set to improve the recognition accuracy of the image detection model for the test set. Among them, the specific implementation of the training of the image detection model for the new training set can refer to the above S502, which will not be repeated here.

[0178] In addition, after the computing device tests the performance of the image detection model, the model training method provided by the present application further includes: the end device collects a generalized sample image that is not from the same source as the sample image of the above-mentioned object, and sends the generalized sample image to the computing device. The computing device can test the generalization performance of the image detection model based on the generalized sample image collected from the end device to detect the generalization ability of the image detection model.

[0179] As described above regarding the format of the image detection model converted by the computing device, the terminal device can deploy the converted image detection model. However, when the image detection model is used to perform the detection and recognition tasks of the object to be detected, the detection results output by the terminal device may have problems such as local feature dependence, misidentification or omission of positive sample categories, and overlap of multiple recognition frames of a single sample in the same frame scene. In other words, the terminal device can reduce the possibility of the above problems by presetting auxiliary strategies during the detection and recognition tasks of the object to be detected. In view of this, Figure 6 As shown, the model training method provided in the present application also includes the following S601 to S604.

[0180] S601: The computing device sends the image detection model in the target format to the terminal device. Correspondingly, the terminal device receives the image detection model in the target format.

[0181] The target format may be the above-mentioned .onnx format.

[0182] S602: The terminal device obtains the video to be detected.

[0183] The video to be detected is the video shot by the terminal device on the object to be detected.

[0184] In one scenario, the video to be detected acquired by the terminal device and the video of multiple objects included in the dark environment captured by the terminal device in S301 are videos from different sources. That is, the image detection model deployed in the terminal device is not trained for the video to be detected.

[0185] S603: The terminal device inputs the videos to be detected into the image detection model respectively to obtain one or more detection results.

[0186] The detection result includes a detection frame corresponding to the object to be detected.

[0187] As a possible implementation, the implementation process of S602 may be as follows: the terminal device inputs the video to be detected into the image detection model. The image detection model detects and identifies the object to be detected appearing in the video to be detected, and determines the recognition result. The image detection model outputs one or more detection results of the object to be detected based on the recognition result.

[0188] In one scenario, the image detection model detects and identifies the object to be detected that appears in the video to be detected, and surrounds the object to be detected with an identification box to determine the position of the object to be detected in the video to be detected. If the image detection model outputs multiple detection results for the target image, that is, the image detection video outputs multiple identification boxes for the object to be detected.

[0189] S604: The terminal device screens one or more detection results based on a preset auxiliary screening strategy to obtain a target detection result.

[0190] As a possible implementation, the preset auxiliary screening strategy may include at least one of the following: a continuous frame auxiliary judgment strategy, a non-maximum suppression strategy, or a confidence filtering strategy.

[0191] The three strategies are described below.

[0192] 1. Continuous frame auxiliary judgment strategy.

[0193] Among them, the continuous frame auxiliary judgment strategy refers to using the detection results of N consecutive frames of images as the target detection results; the offset value between the detection frames of the objects to be detected in the detection results of N frames of images is less than the preset distance; N is a positive integer.

[0194] In one scenario, the image detection model uses a continuous frame auxiliary judgment strategy to determine the target detection result. If the offset value between the recognition frames in the continuous N frames of the image output by the image detection model corresponding to the object to be detected is less than a preset distance, the image detection model determines the recognition frame in the continuous N frames of the image as the target recognition frame. Where N is a positive integer.

[0195] In the present application, the above-mentioned preset distance can be set according to specific needs. For example, the preset distance can be 2 mm, without limitation. Furthermore, in the present application, the offset value between the above-mentioned identification frames can be determined based on the difference in coordinate information of multiple identification frames. For example, taking the determination of the offset value between two identification frames as an example, the offset value between the two identification frames can be determined by the difference between the coordinates of the top corners corresponding to the identification frames. The offset value between the two identification frames can be determined by the difference between the widths or heights corresponding to the identification frames, without limitation.

[0196] In one example, taking N consecutive frames of images corresponding to the object to be detected including image A and image B as an example, if the offset value between the recognition box of the object to be detected in image A and the recognition box of the object to be detected in image B is less than a preset distance, the image detection model determines the recognition box of the object to be detected in image A as the target recognition box of image A, and the image detection model determines the recognition box of the object to be detected in image B as the target recognition box of image B.

[0197] 2. Non-maximum suppression strategy

[0198] Among them, the non-maximum suppression strategy refers to determining the target detection result based on the intersection-over-union ratio and confidence of the detection box of the object to be detected in each frame image.

[0199] In one scenario, the image detection model uses a non-maximum suppression strategy to determine the target detection result. If the image detection model outputs multiple recognition frames in a frame image corresponding to the object to be detected, the image detection model sorts the multiple recognition frames in descending order according to the confidence of each recognition frame, and retains the recognition frames with higher confidence. The image detection model determines the IoU between any two retained recognition frames, and compares the IoU between any two retained recognition frames with a preset IoU threshold to determine the target recognition frame. Among them, the IoU between every two target recognition frames is less than the preset IoU threshold. The above-mentioned preset IoU threshold can be set according to specific needs. For example, the IoU threshold can be 0.5 without restriction.

[0200] In one example, a frame image corresponding to a target image includes a recognition frame A, a recognition frame B, a recognition frame C, and a recognition frame D. If the confidence of recognition frame A is greater than the confidence of recognition frame B, the confidence of recognition frame B is greater than the confidence of recognition frame C, and the confidence of recognition frame C is greater than the confidence of recognition frame D. In addition, the IoU between recognition frame A and recognition frame B is less than a preset IoU threshold, the IoU between recognition frame A and recognition frame C is less than a preset IoU threshold, and the IoU between recognition frame B and recognition frame C is greater than a preset IoU threshold, then the image detection model determines recognition frame A and recognition frame B as the target recognition frame in a frame image corresponding to the target image.

[0201] 3. Confidence filtering strategy

[0202] The confidence filtering strategy refers to filtering detection results whose confidence is less than a preset threshold.

[0203] In one scenario, the image detection model uses a confidence filtering strategy to determine the target detection result. If the image detection model outputs multiple recognition frames in a frame image corresponding to the object to be detected, the image detection model determines the confidence of each recognition frame in the multiple recognition frames, and compares the confidence of each recognition frame in the multiple recognition frames with a preset threshold to determine multiple comparison results. The image detection model retains the recognition frames with a confidence greater than the preset threshold.

[0204] In the present application, the above-mentioned preset threshold can be set as needed. For example, the above-mentioned preset threshold can be 0.7 without limitation.

[0205] In one example, taking a frame image corresponding to the object to be detected including recognition frame A and recognition frame B as an example, if the confidence of recognition frame A is greater than a preset threshold and the confidence of recognition frame B is less than the preset threshold, the image detection model will determine recognition frame A as the target recognition frame of the frame image corresponding to the object to be detected.

[0206] The above implementation process of using three preset auxiliary screening strategies to screen target detection results can be combined arbitrarily, and the embodiments of the present application do not impose any restrictions on this.

[0207] It is understandable that the above-mentioned model training method can be implemented by a model training device. In order to realize the above-mentioned functions, the model training device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the embodiments disclosed in this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments disclosed in this application.

[0208] The embodiments disclosed in the present application can divide the functional modules according to the model training device generated by the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments disclosed in the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0209] Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. Figure 7 As shown, the model training device 70 can be used to perform Figure 3-Figure 6 The model training method shown. The model training device 70 includes: a communication unit 701 and a processing unit 702.

[0210] The communication unit 701 is used to obtain a sample image of an object in a dark environment; the processing unit 702 is used to extract initial feature information of the object in the sample image based on a first convolutional network; wherein the first convolutional network is a convolutional network after the activation value is binarized; the processing unit 702 is also used to adjust the network parameters of the first convolutional network to obtain a second convolutional network when the similarity between the initial feature information and the actual feature information of the object is less than a first preset threshold; the processing unit 702 is also used to train the second convolutional network based on the sample image to obtain an image detection model.

[0211] In one possible implementation, the processing unit 702 is specifically used to perform gradient calculation on the weight value of the first convolutional network based on the loss function corresponding to the similarity to obtain the gradient value; the processing unit 702 is specifically used to adjust the network parameters of the first convolutional network based on the gradient value to obtain the second convolutional network.

[0212] In one possible implementation, the processing unit 702 is specifically used to binarize the weight values ​​of each convolutional layer in the second convolutional network to obtain a processed second convolutional network; the processing unit 702 is specifically used to train the processed second convolutional network based on the sample image to obtain an image detection model.

[0213] In one possible implementation, after binarizing the weight values ​​of each convolutional layer in the second convolutional network to obtain the processed second convolutional network, the processing unit 702 is further used to adjust the weight values ​​of each convolutional layer based on the scaling factor so that the similarity between the feature information extracted from the sample image by the second convolutional network and the actual feature information of the object is greater than a second preset threshold.

[0214] In one possible implementation, the communication unit 701 is also used to obtain a video to be detected; the video to be detected is shot of the object to be detected in a dark environment; the processing unit 702 is also used to input the video to be detected into the image detection model respectively to obtain one or more detection results; wherein the detection result includes a detection box corresponding to the object to be detected; the processing unit 702 is also used to filter one or more detection results based on a preset auxiliary filtering strategy to obtain a target detection result.

[0215] In one possible implementation, the preset auxiliary screening strategy includes at least one of the following: a continuous frame auxiliary judgment strategy; the continuous frame judgment strategy refers to taking the detection results of N consecutive frames of images as the target detection results; the offset value between the detection frames of the objects to be detected in the detection results of N frames of images is less than a preset distance; N is a positive integer; a non-maximum suppression strategy; the non-maximum suppression strategy refers to determining the target detection results based on the intersection-and-union ratio and confidence of the detection frames of the objects to be detected in each frame of the image; a confidence filtering strategy; the confidence filtering strategy refers to filtering the detection results whose confidence is less than a preset threshold.

[0216] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0217] The present disclosure also provides a computer-readable storage medium having instructions stored thereon. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the model training method provided by the above-mentioned embodiment of the present disclosure.

[0218] The embodiments of the present disclosure also provide a computer program product containing instructions, which, when executed on an electronic device, enables the electronic device to execute the model training method provided by the embodiments of the present disclosure.

[0219] Among them, the computer readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above, or any other form of computer readable storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). In the embodiments of the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0220] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Acquire sample images of objects in a dim light environment; Extracting initial feature information of the object in the sample image based on the first convolutional network; The first convolutional network is a convolutional network after activation value binarization; When the similarity between the initial feature information and the actual feature information of the object is less than a first preset threshold, adjusting the network parameters of the first convolutional network to obtain a second convolutional network; The sample image is trained based on the second convolutional network to obtain an image detection model.

2. The method according to claim 1, characterized in that The adjusting the network parameters of the first convolutional network to obtain a second convolutional network includes: Based on the loss function corresponding to the similarity, performing gradient calculation on the weight value of the first convolutional network to obtain a gradient value; Based on the gradient value, the network parameters of the first convolutional network are adjusted to obtain the second convolutional network.

3. The method according to claim 1 or 2, characterized in that: The step of training the sample image based on the second convolutional network to obtain an image detection model includes: Binarizing the weight values ​​of each convolutional layer in the second convolutional network to obtain a processed second convolutional network; The processed second convolutional network is trained based on the sample image to obtain the image detection model.

4. The method according to claim 3, characterized in that After binarizing the weight values ​​of each convolutional layer in the second convolutional network to obtain the processed second convolutional network, the method further includes: Based on the scaling factor, the weight value of each convolutional layer is adjusted so that the similarity between the feature information extracted from the sample image by the second convolutional network and the actual feature information of the object is greater than a second preset threshold.

5. The method according to claim 1, characterized in that The method further comprises: Obtain a video to be detected; the video to be detected is a video shot of the object to be detected; Input the videos to be detected into the image detection model respectively to obtain one or more detection results; the detection results include detection boxes corresponding to the objects to be detected; The one or more detection results are screened based on a preset auxiliary screening strategy to obtain a target detection result.

6. The method according to claim 5, characterized in that The preset auxiliary screening strategy includes at least one of the following: Continuous frame auxiliary judgment strategy; the continuous frame judgment strategy refers to using the detection results of N consecutive frames of images as the target detection results; the offset value between the detection frames of the object to be detected in the detection results of the N frames of images is less than a preset distance; N is a positive integer; Non-maximum suppression strategy; The non-maximum suppression strategy refers to determining the target detection result based on the intersection-over-union ratio and confidence of the detection frame of the object to be detected in each frame of the image; Confidence filtering strategy: The confidence filtering strategy refers to filtering detection results whose confidence is less than a preset threshold.

7. A model training device, characterized in that: The model training device comprises: a communication unit and a processing unit; The communication unit is used to obtain a sample image of an object in a dark light environment; The processing unit is used to extract initial feature information of the object in the sample image based on a first convolutional network; the first convolutional network is a convolutional network after activation values ​​are binarized; The processing unit is further configured to adjust the network parameters of the first convolutional network to obtain a second convolutional network when the similarity between the initial feature information and the actual feature information of the object is less than a first preset threshold; The processing unit is further used to train the sample image based on the second convolutional network to obtain an image detection model.

8. The device according to claim 7, characterized in that The processing unit is specifically configured to perform gradient calculation on the weight value of the first convolutional network based on the loss function corresponding to the similarity to obtain a gradient value; The processing unit is specifically used to adjust the network parameters of the first convolutional network based on the gradient value to obtain the second convolutional network.

9. The device according to claim 7 or 8, characterized in that The processing unit is specifically used to perform binarization processing on the weight value of each convolutional layer in the second convolutional network to obtain a processed second convolutional network; The processing unit is specifically used to train the processed second convolutional network based on the sample image to obtain the image detection model.

10. The device according to claim 9, characterized in that The processing unit is further used to adjust the weight value of each convolution layer based on the scaling factor so that the similarity between the feature information extracted from the sample image by the second convolution network and the actual feature information of the object is greater than a second preset threshold.

11. The device according to claim 7 or 8, characterized in that The communication unit is further used to obtain a video to be detected; the video to be detected is a video shot of the object to be detected; The processing unit is further used to input the videos to be detected into the image detection model respectively to obtain one or more detection results; the detection results include a detection frame corresponding to the object to be detected; The processing unit is further used to screen the one or more detection results based on a preset auxiliary screening strategy to obtain a target detection result.

12. The device according to claim 11, characterized in that The preset auxiliary screening strategy includes at least one of the following: a continuous frame auxiliary judgment strategy; the continuous frame judgment strategy refers to using the detection results of N consecutive frames of images as the target detection results; the offset value between the detection frames of the object to be detected in the detection results of the N frames of images is less than a preset distance; N is a positive integer; Non-maximum suppression strategy; The non-maximum suppression strategy refers to determining the target detection result based on the intersection-over-union ratio and confidence of the detection frame of the object to be detected in each frame of the image; Confidence filtering strategy: The confidence filtering strategy refers to filtering detection results whose confidence is less than a preset threshold.

13. A model training device, characterized in that: include: Memory and processor; The memory is coupled to the processor; The memory is used to store instructions executable by the processor; When the processor executes the instructions, it performs the model training method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the model training method as described in any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product includes computer program instructions, which, when executed by a processor, implement the model training method as described in any one of claims 1 to 6.

Images