Processing method of image target detection model, image target detection method and device

Abstract

The present disclosure relates to a processing method of an image target detection model, an image target detection method and equipment, and relates to the technical field of computers. The processing method comprises determining quantifiable processing nodes in the image target detection model, wherein the value range of feature data output by the quantifiable processing nodes is in the same value range; and performing format conversion and quantization processing of node parameters on the quantifiable processing nodes, so as to convert the format of the image target detection model into a model format that can be called by a chip. In this way, since the nodes subjected to quantization processing are all in the same value range, when the quantization processing is performed, the model data can obtain more accurate output results, and the precision loss caused in the model quantization process is reduced. The more accurate quantization output results can improve the precision of the image target detection model in target detection and reduce errors.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a processing method for an image target detection model, an image target detection method, and an apparatus. Background Technology

[0002] Object detection is one of the most fundamental problems in computer vision, aiming to identify objects in images or videos and determine their specific locations. While image object detection models built using deep learning techniques have been widely applied in scenarios such as autonomous driving and smart cities, the problem of relatively low detection accuracy still exists in these technologies. Summary of the Invention

[0003] To address the aforementioned issues, this disclosure provides a processing method for an image target detection model, an image target detection method, and an apparatus.

[0004] The first aspect of this disclosure provides a processing method for an image object detection model, comprising:

[0005] In the image target detection model, quantizable nodes are identified, and the value range of the feature data output by the quantizable nodes is within the same value range.

[0006] The quantizable processing nodes are subjected to format conversion and node parameter quantization to convert the format of the image target detection model into a chip-callable model format. The node parameters of the quantizable processing nodes in the image target detection model after format conversion and quantization are integers.

[0007] Optionally, the image target detection model includes multiple intermediate layers, and the step of determining the quantizable nodes in the image target detection model includes:

[0008] The image target detection model is invoked to perform target detection on the sample image, and the first output data after target detection is obtained for each node;

[0009] The image target detection model is quantized.

[0010] The quantized image target detection model is then used to perform target detection on the sample image.

[0011] Each intermediate layer obtains feature data after performing target detection on the sample image, and obtains the second output data of each node after performing target detection.

[0012] Based on the first output data and the second output data, the quantizable processing nodes in the image target detection model are determined.

[0013] Optionally, the step of calling the quantized image target detection model to perform target detection on the sample image and obtaining the second output data after target detection for each node includes:

[0014] Determine whether the value range of the feature data corresponding to each intermediate layer is within the same value range;

[0015] The second output data after all nodes have performed target detection is obtained from the intermediate layer where the value range of the feature data is within the same value range.

[0016] Optionally, the step of determining the quantizable processing node in the image target detection model based on the first output data and the second output data includes: determining the precision deviation between the second output data and the first output data, and confirming that the node with the precision deviation less than a preset deviation is the quantizable processing node.

[0017] Optionally, the method further includes:

[0018] Nodes whose accuracy deviation is less than a preset deviation are identified as unquantifiable nodes.

[0019] Optionally, the steps for obtaining the first output data after object detection at each node include:

[0020] Each node obtains the first target coordinate value, first category probability, or first confidence level for the target detection in the sample image.

[0021] Optionally, the step of obtaining the second output data after target detection for all nodes in the intermediate layer whose value ranges are within the same value range of the feature data includes:

[0022] The value range of the feature data is obtained in the middle layer where the value range is the same, and each node obtains the second target coordinate value, second category probability, or second confidence level for the target detection in the sample image.

[0023] Optionally, the step of determining the precision deviation between the second output data and the first output data, and confirming that nodes with a precision deviation less than a preset deviation are quantifiable processing nodes, includes:

[0024] Nodes whose deviation between the first target coordinate value and the second target coordinate value is less than a preset coordinate value deviation are identified as quantifiable processing nodes; or

[0025] Nodes whose deviation between the first category probability and the second category probability is less than a preset category probability deviation are identified as quantifiable processing nodes; or

[0026] Nodes whose deviation between the first confidence level and the second confidence level is less than a preset confidence level deviation are identified as quantifiable nodes.

[0027] Optionally, the steps of format conversion and node parameter quantization of the quantizable nodes include:

[0028] Multiple pre-obtained feature images are used as input images for the quantization process, and the quantizable processing nodes are quantized. The multiple feature images include multiple brightness feature images, multiple scene feature images, and / or multiple illumination feature images.

[0029] A second aspect of this disclosure provides an image target detection method, comprising:

[0030] An image target detection model is invoked to perform target detection on the image to be detected, thereby obtaining sensitive targets in the image to be detected, wherein the image target detection model is processed by the processing method described in any one of the first aspects.

[0031] Optionally, the step of calling an image target detection model to perform target detection on the image to be detected and obtaining sensitive targets in the image to be detected includes:

[0032] Error and redundant borders in the bounding boxes obtained by target detection of the image to be detected are removed so that each sensitive target is selected by an initial target box;

[0033] When the image to be detected includes multiple video frames, the final target bounding box of the sensitive target in the current video frame is determined based on the initial target bounding box on the current video frame and the initial target bounding box on the previous video frame.

[0034] Optionally, the step of removing erroneous and redundant borders from the bounding boxes obtained by target detection of the image to be detected includes:

[0035] Calculate the confidence score and class probability of each bounding box obtained by target detection on the image to be detected;

[0036] Among the bounding boxes with a confidence level greater than a preset confidence threshold, the bounding box with the highest category probability is used as the initial target box for the sensitive target.

[0037] Optionally, when the image to be detected includes multiple video frames, the step of determining the final target bounding box of the sensitive target in the current video frame based on the initial target bounding box on the current video frame and the initial target bounding box on the previous video frame includes:

[0038] Calculate the overlap area between the initial target bounding box on the current video frame and the initial target bounding box on the previous video frame;

[0039] If the overlap area between the initial target box on the current video frame and the initial target box on the previous video frame is greater than a preset overlap threshold, calculate the weight information of the initial target box on the current video frame and the weight information of the initial target box on the previous video frame.

[0040] Based on the position information of the initial target box in the current video frame, the position information of the initial target box in the previous video frame, the weight information of the initial target box in the current video frame, and the weight information of the initial target box in the previous video frame, position weight information is calculated;

[0041] A new bounding box is established based on the location-weighted information, which serves as the final target bounding box for the sensitive target in the current video frame.

[0042] Optionally, the method includes: performing the following steps in parallel:

[0043] After obtaining the image to be detected, the image to be detected is converted and quantized, and the converted and quantized image to be detected is cached.

[0044] The image target detection model is invoked to perform target detection on the cached image to be detected, thereby obtaining the sensitive targets in the image to be detected.

[0045] Optionally, after the step of calling the image target detection model to perform target detection on the image to be detected and obtaining the sensitive targets in the image to be detected, the method further includes:

[0046] The sensitive targets are desensitized.

[0047] A third aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the processing method of the image target detection model described in the first aspect or the image target detection method described in the second aspect.

[0048] A fourth aspect of this disclosure provides an electronic device comprising:

[0049] A memory on which computer programs are stored;

[0050] A processor is configured to execute the computer program in the memory to implement the image target detection model processing method of the first aspect or the image target detection method of the second aspect.

[0051] The fifth aspect of this disclosure provides a processing apparatus for an image target detection model, comprising:

[0052] The filtering module is configured to determine quantizable nodes in the image target detection model, wherein the value range of the feature data output by the quantizable nodes is within the same value range.

[0053] The processing module is configured to perform format conversion and node parameter quantization on the quantizable processing node to convert the format of the image target detection model into a chip-callable model format, wherein the node parameters of the quantizable processing node after format conversion and quantization are integers.

[0054] A sixth aspect of this disclosure provides an image target detection apparatus, comprising:

[0055] The detection module is configured to call an image target detection model to perform target detection on the image to be detected, and obtain the sensitive targets in the image to be detected, wherein the image target detection model is processed by the above-described processing method.

[0056] The above technical solution avoids directly quantizing all nodes in an image target detection model during quantization. Instead, it first selects quantizable nodes and then quantizes them. Because the quantized nodes are all within the same value range, the model can produce more accurate output results. This effectively avoids quantizing floating-point parameters with different value ranges, which could lead to some floating-point parameters being quantized to zero. This reduces accuracy loss during quantization and improves the accuracy of the image target detection model, thus lowering errors.

[0057] Furthermore, since the parameters of a quantized node are converted from floating-point to integer, the quantized node can utilize the resources of an integer-type computing chip for node processing. Compared to floating-point computing chips, integer-type computing chips can improve the processing speed of the node, thereby improving the image detection speed.

[0058] In summary, the above technical solutions can effectively balance the detection efficiency and accuracy of image target detection models.

[0059] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0060] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0061] Figure 1A flowchart illustrating an image target detection model processing method provided as an exemplary embodiment of this disclosure;

[0062] Figure 2 A flowchart of the steps for determining a quantifiable processing node provided as an exemplary embodiment of this disclosure;

[0063] Figure 3 A flowchart of the steps for determining second output data provided as an exemplary embodiment of this disclosure;

[0064] Figure 4 A flowchart of an image target detection method provided as an exemplary embodiment of this disclosure;

[0065] Figure 5 A flowchart illustrating the steps for removing redundant and erroneous borders provided in an exemplary embodiment of this disclosure;

[0066] Figure 6 A flowchart of the steps for determining a final target box provided as an exemplary embodiment of this disclosure;

[0067] Figure 7 A schematic diagram illustrating an application example of an image target detection method on an in-vehicle infotainment platform, provided as an exemplary embodiment of this disclosure;

[0068] Figure 8 A block diagram of a processing apparatus for an image target detection model provided as an exemplary embodiment of the present disclosure;

[0069] Figure 9 A block diagram of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0070] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0071] With the rapid development of the Internet of Things (IoT) and mobile computing technologies, an increasing number of smart terminals are connecting to the internet as edge devices, generating massive amounts of real-time data. Leveraging the powerful computing capabilities of cloud computing centers, this data can be processed and analyzed using intelligent computing technologies such as deep learning, thereby building real-time intelligent applications and obtaining more useful information or making more effective decisions. However, the inherent end-to-end transmission bottleneck of the internet severely impacts the real-time performance of such intelligent applications. In recent years, edge computing has emerged as a highly promising computing paradigm, directly processing some or all of the data on the edge devices that generate it, without needing to upload all the data to a remote cloud platform.

[0072] Object detection is a crucial application of computer vision technology. Image object detection models built based on deep learning algorithms can be applied across various computing platforms. With the rapid popularization of intelligent vehicles, in-vehicle infotainment platforms also need to perform tasks based on image object detection models. For example, they can detect sensitive targets in videos or images captured by cameras in intelligent vehicles, and then de-identify these sensitive targets to prevent the leakage of personal privacy information. When deploying de-identification algorithms based on image object detection models on in-vehicle infotainment platforms, the algorithm model can first be trained on a server. Then, a conversion tool can be used to convert the model to the format corresponding to the chips on the in-vehicle infotainment platform. Next, a Software Development Kit (SDK) can be used to call the model to detect videos or images, obtain the output results, and then perform post-processing to identify sensitive targets, thus completing the de-identification of the videos or images.

[0073] To accelerate model deployment, chips such as Digital Signal Processing (DSP) can be used for faster computation. DSP chips require data quantization during computation. The inventors discovered that when quantizing image target detection models, the model data needs to be converted from floating-point to integer types. However, the same node data in an image target detection model may have multiple numerical ranges. During quantization, some non-zero data may be incorrectly quantized to zero. For example, suppose the output of an image target detection model has two numerical ranges: one part ranges from 0 to 1, and the other ranges from 0 to hundreds or thousands. When quantizing these two different ranges, data in the 0-1 range will be quantized to 0, resulting in non-zero data being quantized to zero. This erroneous quantization process leads to errors in target detection results. For example, some targets in an image or video may not be detected correctly, or there may be jitter in the detection of the same target in consecutive frames of a video.

[0074] In view of this, embodiments of this disclosure provide an image target detection model processing method, such as... Figure 1 The diagram shows a flowchart of an image target detection model processing method provided in an exemplary embodiment of this disclosure. The image target detection model processing method includes the following steps S11 to S12.

[0075] In step S11, quantizable nodes in the image target detection model are determined.

[0076] In practical use, the image target detection model in this disclosure can perform target detection on images or videos. Since videos are also composed of consecutive video frames, detecting targets in a video means detecting targets in each video frame. This disclosure does not limit the application scenarios of the image target detection model.

[0077] The image target detection model in this disclosure can be based on a deep learning network architecture. This model may include multiple intermediate layers, each of which may include multiple nodes, and each node can perform target detection. For example, each node can detect targets such as faces and license plates in videos or images. Alternatively, each node can only detect one type of target; for example, node A can only detect faces, while node B can detect license plates. This disclosure does not limit the specific architecture of the image target detection model.

[0078] Since different nodes can be configured with different model parameters, these different nodes may produce different quantization results after quantization. Although all nodes can be quantized, some nodes are actually unsuitable for quantization. If these unsuitable nodes are quantized, unavoidable errors will be introduced. By screening suitable nodes for quantization and excluding unsuitable nodes, errors can be avoided in the image target detection model. In this embodiment, quantizable nodes are those whose feature data value ranges are within the same range. Nodes with different feature data value ranges are determined to be unquantizable nodes. Only data with the same value range will prevent non-zero data from being quantized to zero after quantization.

[0079] In step S12, the format of the quantizable nodes is converted and the node parameters are quantized to convert the format of the image target detection model into a chip-callable model format.

[0080] After identifying quantizable and non-quantizable processing nodes, these nodes can be processed separately. In this embodiment, the node names of quantizable processing nodes can be configured in the model conversion parameters, converting the parameters of these nodes into a chip-readable format. Simultaneously, the parameters of quantizable processing nodes can be quantized, converting floating-point data into integer data. For non-quantizable processing nodes, no data quantization is performed; only format conversion is applied. By operating on quantizable and non-quantizable processing nodes separately, errors introduced into the image target detection model can be avoided.

[0081] It is understandable that the quantization process converts data from floating-point to integer. This conversion process can be accelerated by a DSP chip. In the image target detection model after format conversion and quantization, the node parameters of the quantizable nodes are integers, while the node parameters of the non-quantizable nodes in the image target detection model after format conversion include floating-point values.

[0082] The above technical solution avoids directly quantizing all nodes in an image target detection model during quantization. Instead, it first selects quantizable nodes and then quantizes them. Because the quantized nodes are all within the same value range, the model can produce more accurate output results. This effectively avoids quantizing floating-point parameters with different value ranges, which could lead to some floating-point parameters being quantized to zero. This reduces accuracy loss during quantization and improves the accuracy of the image target detection model, thus lowering errors.

[0083] Furthermore, since the parameters of a quantized node are converted from floating-point to integer, the quantized node can utilize the resources of an integer-type computing chip for node processing. Compared to floating-point computing chips, integer-type computing chips can improve the processing speed of the node, thereby improving the image detection speed.

[0084] In summary, the above technical solutions can effectively balance the detection efficiency and accuracy of image target detection models.

[0085] In some embodiments of this disclosure, the suitability of a node for quantization can be determined by the deviation between the output results of the node before and after target detection. For example... Figure 2 The flowchart shown is a step for determining a quantizable processing node according to an exemplary embodiment of the present disclosure. The step for determining a quantizable processing node includes the following steps S121 to S124.

[0086] Step S121: Call the image object detection model to perform object detection on the sample image and obtain the first output data after object detection at each node.

[0087] Sample images can be selected according to actual needs. Sample images can be individual image data or video frames from a video. Sample images can include multiple targets to be detected, and the number and form of the targets to be detected can be set according to the actual situation of the image target detection model.

[0088] In some embodiments, when the image target detection model performs target detection, it may obtain the first target coordinate value, first class probability, or first confidence score of each node for the target detection in the sample image. It is understood that if the sample image includes multiple targets to be detected, then each detected target can have its own first target coordinate value, first class probability, or first confidence score.

[0089] In step S122, the image target detection model is quantized.

[0090] After using an image target detection model that has not undergone quantization for target detection, the image target detection model can be quantized again, that is, the parameters and other data configured for each node in the image target detection model can be quantized. In the embodiments of this disclosure, the quantization process can be symmetric quantization, asymmetric quantization, or other forms of quantization methods, and the embodiments of this disclosure do not limit the form of quantization processing.

[0091] In step S123, the quantized image target detection model is called to perform target detection on the sample image, and the second output data after target detection of each node is obtained.

[0092] After quantizing the image object detection model—that is, quantizing the parameters and other data configured for each node—the quantized image object detection model is then called to detect the same sample image. This yields the feature data output by the quantized intermediate layer after object detection. The data obtained by the quantized image object detection model in detecting objects in the sample image is then used as the second output data.

[0093] In step S124, quantizable processing nodes in the image target detection model are determined based on the first output data and the second output data.

[0094] The second output data of a node can be compared with the first output data before quantization to determine whether the node is suitable for quantization. By comparing the difference between the first and second output data, it can be determined whether the node has experienced any deviation after quantization. More specifically... Figure 3 This is a flowchart illustrating the steps of determining a quantifiable processing node according to an exemplary embodiment. Figure 3 As shown, the step of calling the quantized image target detection model to perform target detection on the sample image and obtaining the second output data after target detection at each node may include steps S131 to S133.

[0095] In step S131, feature data obtained by each intermediate layer after performing target detection on the sample image are obtained respectively.

[0096] After the image target detection model has undergone quantization, it can obtain the feature data of each intermediate layer for target detection of the sample image.

[0097] In step S132, it is determined whether the value range of the feature data corresponding to each intermediate layer is within the same value range.

[0098] After detecting the sample image, the output data of the intermediate layer can be the feature data obtained after target detection of the sample information. Determining whether the value range of the feature data output by each intermediate layer is consistent determines whether these feature data fall within the same value range. If the feature data output by the intermediate layers are within the same value range, then this intermediate layer is suitable for quantization processing. If the value range of the feature data output by a certain intermediate layer is inconsistent, then this intermediate layer is not suitable for quantization processing, and its processing can be postponed to the post-processing stage. For example, if the feature data output by a certain intermediate layer is a value between 0 and 1, excluding values ​​from 0 to hundreds or thousands, then it can be determined that the value range of the feature data of this intermediate layer is within the same value range.

[0099] In step S133, the second output data after target detection is obtained for all nodes in the intermediate layer whose feature data value range is within the same value range.

[0100] Once the value range of the feature data is determined to be within the same intermediate layer, the second output data obtained by all nodes in this intermediate layer performing target detection can be obtained. As mentioned earlier, the first output data is obtained when the image target detection model has not undergone quantization processing; that is, the first output data is obtained after the nodes that have not undergone quantization perform target detection on the sample image. After the image target detection model has undergone quantization processing, and the node data has also undergone quantization processing, the second output data is obtained after performing target detection on the same sample image. It can be understood that the first output data may include target coordinate values, class probabilities, or confidence scores, etc. Therefore, in order to facilitate the comparison of the deviation of the node output data before and after quantization, the second output data may also be the second target coordinate values, second class probabilities, or second confidence scores obtained by each node for target detection in the sample image. The embodiments of this disclosure do not limit the categories of the first and second output data.

[0101] In one possible implementation, the step of determining the quantizable processing node in the image target detection model based on the first output data and the second output data may include step S141.

[0102] In step S141, the precision deviation between the second output data and the first output data is determined, and the nodes with a precision deviation less than a preset deviation are confirmed as quantifiable processing nodes.

[0103] Furthermore, after identifying the intermediate layer suitable for quantization, since an intermediate layer includes multiple nodes, not all nodes are suitable for quantization. Further screening of suitable nodes is required. In this embodiment, the suitability of a node for quantization is determined by comparing the node's second output data with its first output data before quantization. By comparing the difference between the first and second output data, it can be determined whether the node has experienced any deviation after quantization.

[0104] It is understood that in the embodiments of this disclosure, the first output data and the second output data being compared refer to the output data of the same node. Only by comparing the output data of the same node before and after quantization can it be determined whether the output data of that node has introduced errors during the quantization process.

[0105] In another possible implementation, the processing method of the image target detection model may further include the following steps S142 and S143.

[0106] In step S142, nodes with an accuracy deviation greater than or equal to a preset deviation are determined to be nodes that cannot be quantized.

[0107] In step S143, the format of the unquantizable processing nodes is converted.

[0108] In this embodiment, nodes whose precision deviation between the second output data and the first output data is less than a preset deviation are identified as quantizable processing nodes. Nodes whose precision deviation is greater than or equal to the preset deviation are identified as non-quantizable processing nodes. Quantizable processing nodes can then undergo subsequent quantization processing, while non-quantizable processing nodes can only undergo format conversion. In a possible implementation, quantization processing of quantizable processing nodes can be performed using a chip with faster integer processing speed, while non-quantizable processing nodes can be processed using a chip suitable for floating-point operations. This avoids errors caused by using a chip with faster integer processing speed to process non-quantizable processing nodes.

[0109] In this embodiment, when determining whether a node is suitable for quantization, the system first identifies intermediate layers in the image target detection model where the output feature data falls within the same value range. Then, it compares the output data of nodes in these intermediate layers, comparing the output data obtained before and after quantization for target detection of the same sample image to determine if a node is suitable for quantization. Nodes whose output data deviation before and after quantization is less than a preset deviation are identified as quantizable nodes. Even after quantization, these nodes achieve higher detection accuracy because the quantization process does not introduce errors into them. Nodes that are not quantizable are not quantized, thus avoiding errors introduced by the quantization process. This embodiment, by screening and classifying nodes in the image target detection model, minimizes the deviations introduced by the quantization process. When this image target detection model runs on a chip with faster integer processing, it leverages the chip's accelerated computation to improve target detection efficiency, resulting in more accurate detection results and smaller detection errors.

[0110] In some embodiments, it can be determined whether there is a deviation in the data before and after quantization of a node by judging whether there is a deviation in different types of data. Specifically, the step of determining the precision deviation between the second output data and the first output data, and confirming that the node whose precision deviation is less than a preset deviation is a quantizable processing node includes the following steps S1261, S1262, or S1263.

[0111] In step S1261, nodes whose deviation between the first target coordinate value and the second target coordinate value is less than a preset coordinate value deviation are determined to be quantifiable processing nodes.

[0112] In step S1262, nodes whose deviation between the first category probability and the second category probability is less than a preset category probability deviation are identified as quantifiable processing nodes.

[0113] In step S1263, nodes whose deviation between the first confidence level and the second confidence level is less than a preset confidence level deviation are identified as quantifiable processing nodes.

[0114] It is understandable that steps S1261, S1262 and S1263 do not need to be satisfied simultaneously. Determining a node as a quantifiable processing node can be done by satisfying just one of the conditions.

[0115] For example, before quantization, a node performs target detection on a sample image, obtaining the target coordinates of a target T, i.e., the first target coordinates. After quantization, the node performs target detection again on the sample image, obtaining the second target coordinates of target T. In this embodiment, after target detection using an image target detection model, a bounding box is generated on the detected image to select a target. The first and second target coordinates can both be the coordinates of the bounding box in a coordinate system established based on the image; that is, the first and second target coordinates are the coordinates of the bounding box obtained after detecting the same target in the same coordinate system. By comparing these two target coordinates, if the deviation between the second and first target coordinates exceeds a preset deviation, then the node can be considered unsuitable for quantization because quantization increases the error of target detection performed by the node. Conversely, the node can be considered suitable for quantization because quantization does not increase the error of target detection performed by the node.

[0116] The determination of whether a node is quantifiable can also be based on category probability or confidence level. After completing target detection, each node can output the coordinates of the bounding box of the selected target, as well as the category probability and confidence level of the target. These two data points can also be used to determine whether there is any bias in the node's output data. If the deviation between these two data points before and after quantization is large, exceeding the preset category probability deviation or preset confidence level deviation, then the quantization process can be considered to have introduced a large error into the node, and the node is considered unsuitable for quantization.

[0117] This disclosure does not limit the data selection for determining quantizable nodes. Target coordinates, category probability, and confidence level can be used as the criterion; a node is considered quantizable if the deviation of any one of these data points is less than a preset deviation. Alternatively, two or three data points can be used as the criterion; a node is considered quantizable only if the deviations of all three data points are less than the preset deviation. Nodes selected in this way are less affected by quantization, resulting in more accurate output. The precision deviation between the first and second output data is used as the criterion for determining whether a node is suitable for quantization. Nodes determined to be suitable for quantization can be quantized during subsequent model conversion and quantization. Nodes unsuitable for quantization can be processed in the post-processing stage. In the post-processing stage, unsuitable intermediate layers and nodes can be processed by the central processing unit (CPU). Since the architecture of the CPU differs from that of the DSP chip, it is not necessary to convert the data in the image target detection model into integers before processing. Therefore, processing unsuitable intermediate layers and nodes by the CPU will not produce errors.

[0118] In practical applications, after identifying suitable nodes for quantization, a conversion tool can be used to convert the image target detection model to formats such as ONNX (Open Neural Network Exchange). This conversion process can convert both quantizable and non-quantizable nodes. After conversion, quantizable nodes can be quantized using a quantization tool, while non-quantizable nodes remain unquantized. This allows the converted and quantized image target detection model to run efficiently and quickly on chips with faster integer processing capabilities (such as DSP chips). The quantization process can be either symmetric or asymmetric quantization as needed; this embodiment does not limit the specific quantization method.

[0119] Simultaneously, during the quantization process, multiple pre-obtained feature images can be used as input images for the quantizable processing node. These multiple feature images can include multiple brightness feature images, multiple scene feature images, and / or multiple illumination feature images. The multiple brightness feature images can be images with identical display content and display parameters other than brightness. Different brightness feature images differ only in brightness, with all other parameters being the same. Similarly, multiple scene feature images can be images with identical display parameters but displaying different scenes. Different scene feature images display different scenes; for example, one scene feature image can display an image from a certain perspective inside a vehicle, while another scene feature image can display an image from a different perspective inside the vehicle. Multiple illumination feature images can be images with identical display content and display parameters except for illumination parameters. Different illumination feature images can display the same scene, but parameters such as the angle of light illumination and the intensity of light can differ between them. Of course, other types of images can also be used as feature images; this disclosure does not limit the selection of feature images. By quantizing the diversity of the input image data distribution, the generalization ability of the quantized model can be improved.

[0120] This disclosure also provides a target detection method, which includes: calling an image target detection model to perform target detection on an image to be detected, thereby obtaining sensitive targets in the image to be detected. The image target detection model is processed using the above-described processing method.

[0121] In the target detection method of this embodiment, the image target detection model determines whether all nodes are suitable for quantization, thereby increasing the detection speed and accuracy of the target detection results when performing target detection on the image to be detected, thus improving the efficiency of target detection. Without increasing system resources and computing power consumption, it can reduce errors in the target detection process and improve the accuracy and speed of the algorithm. This target detection method, due to its higher detection efficiency and more accurate results, can run accurately and efficiently on the vehicle-mounted system platform. When facing the task of desensitizing videos or images on the vehicle-mounted system platform, it can accurately and efficiently detect the targets that need to be desensitized, achieving efficient and accurate execution of the desensitization task.

[0122] Detailed, such as Figure 4 The diagram shown is a flowchart of an image target detection method provided in an exemplary embodiment of this disclosure. The step of calling an image target detection model to perform target detection on the image to be detected and obtaining sensitive targets in the image to be detected may include the following steps S31 and S32.

[0123] In step S31, erroneous and redundant borders in the bounding boxes obtained from target detection of the image to be detected are removed so that each sensitive target is selected by an initial target box.

[0124] Image object detection models generate different bounding boxes for different targets during object detection, with a single target being selected by multiple bounding boxes simultaneously. These bounding boxes may have different selection ranges, and some may be redundant or even erroneous, with completely incorrect selection ranges. Ultimately, only the bounding box closest to the true target is considered the correct bounding box. Therefore, erroneous and redundant bounding boxes need to be removed, ensuring that each sensitive target retains only one initial bounding box. When the image to be detected is a single image, this initial bounding box can be used as the final bounding box for the sensitive target on that image. When the image to be detected is a video, the initial bounding boxes need further processing to obtain the final bounding boxes for the sensitive targets on each video frame.

[0125] When the image to be detected is a video, the image object detection model performs object detection on multiple video frames separately. At this time, each video frame will have erroneous or redundant borders. By removing erroneous or redundant borders from each video frame, each sensitive target in each video frame will be selected by only one border, and this border is the initial target bounding box of that sensitive target.

[0126] In step S32, when the image to be detected includes multiple video frames, the final target bounding box of the sensitive target on the current video frame is determined based on the initial target bounding box on the current video frame and the initial target bounding box on the previous video frame.

[0127] When the image to be detected is video, each sensitive target in the video frame corresponds to an initial bounding box. Unlike images, video object detection, even in a static video where the sensitive target hasn't moved, will exhibit slight pixel changes between adjacent frames. This can cause the initial bounding boxes in adjacent frames to shift within a certain range, resulting in frame-to-frame jitter. If the sensitive target moves, the pixel changes between adjacent frames become more pronounced, making the frame-to-frame jitter even more significant. This can lead to excessively large shifts in the bounding boxes as the sensitive target moves, resulting in a less smooth detection process and lower visual stability. In this case, if the initial bounding boxes in each video frame are not processed, the frame-to-frame jitter between adjacent frames will negatively impact the viewing experience.

[0128] In this embodiment, the initial target box on the current video frame and the initial target box on the previous video frame both select the same sensitive target. The process of determining whether the initial target boxes in two consecutive frames select the same target can be achieved by calculating the initial target box interaction ratio. For example, firstly, in the current video frame (which is not the first frame), a first initial target box for a certain target is determined. If the processed video frame is the first frame, the operation ends, and a new video frame is extracted from multiple video frames as the current video frame. It is understood that after determining the first initial target box, an initial target box with an interaction ratio (Intersection over Union, IoU) exceeding a certain value can be found in the previous video frame. This initial target box is then used as the second initial target box. Targets selected by initial target boxes with an IoU exceeding a certain value are considered to be the same target as those selected by the first initial target box. This embodiment does not limit the method for determining the same target.

[0129] In this embodiment, a new bounding box is determined on the current video frame based on the initial bounding box on the current video frame and the initial bounding box on the previous video frame. This new bounding box is then used as the final bounding box for the sensitive target. This processing flow can determine the final bounding box for all video frames except the first frame, ensuring that the final bounding boxes of adjacent frames are obtained based on the initial bounding boxes of those two adjacent frames. This eliminates inter-frame jitter, improves the visual stability of the bounding boxes when detecting targets in the video, and enhances the viewing experience.

[0130] In some embodiments, such as Figure 5 The diagram shows a flowchart of the steps for removing redundant and erroneous borders provided in an exemplary embodiment of this disclosure. The steps for removing redundant and erroneous borders may include steps S311 and S312.

[0131] In step S311, the confidence level and class probability of each bounding box obtained by target detection in the image to be detected are calculated respectively.

[0132] Depending on the specific needs, the output data of the image object detection model may include an objectness score (representing the probability of object presence), class probability, bounding box regression parameters, etc. This disclosure does not limit the categories of the output data of the image object detection model; the confidence level can be obtained by multiplying the objectness score and the class probability.

[0133] In step S312, the bounding box with the highest category probability among the bounding boxes with a confidence level greater than a preset confidence threshold is taken as the initial target box of the sensitive target.

[0134] The preset confidence threshold can be determined according to actual needs. Boundaries with a confidence level lower than the preset threshold are considered incorrectly extracted and are filtered out. Among the remaining bounding boxes with a confidence level greater than the preset threshold, the bounding box with the highest class probability is retained as the initial target box. After removing incorrect bounding boxes, a non-maximum suppression (NMS) algorithm can be used to select the bounding box with the highest class probability from the remaining bounding boxes.

[0135] By using confidence and class probability as the basis for removing erroneous and redundant bounding boxes, the next bounding box can be retained from multiple bounding boxes corresponding to a sensitive target in an image or video frame. This bounding box can be the result obtained by the image target detection model after detecting the sensitive target, thus enabling the detection of the sensitive target.

[0136] In some embodiments, such as Figure 6 The diagram shows a flowchart of the steps for determining a final target bounding box according to an exemplary embodiment of this disclosure. The step of determining the final target bounding box of the sensitive target in the current video frame based on the initial target bounding box on the current video frame and the initial target bounding box on the previous video frame may include steps S321 to S324.

[0137] In step S321, the overlap area between the initial target box on the current video frame and the initial target box on the previous video frame is calculated.

[0138] In step S322, if the overlap area between the initial target box on the current video frame and the initial target box on the previous video frame is greater than a preset overlap threshold, the weight information of the initial target box on the current video frame and the weight information of the initial target box on the previous video frame are calculated.

[0139] As described above, in this embodiment of the disclosure, the initial target box on the current video frame and the initial target box on the previous video frame both select the same sensitive target. When the overlap area between the initial target box on the current video frame and the initial target box on the previous video frame is greater than a preset overlap threshold, the weights of the initial target boxes on the two video frames are calculated. Specifically, the weight pre_weight of the initial target box on the previous video frame can be calculated using the following formula.

[0140] pre_weight=time_weight·space_weight

[0141] Wherein, time_weight is the temporal weight of the initial target box on the previous video frame, and space_weight is the spatial weight of the initial target box on the previous video frame. In this embodiment, time_weight can be set to a fixed value less than 1, for example, time_weight = 0.8. This embodiment does not limit the value of time_weight. The spatial weight of the initial target box on the previous video frame, space_weight, can be calculated using the following formula.

[0142] space_weight=exp(-diff·diff / 2)

[0143] Where diff is a Gaussian function, diff = 1 - IoU, where IoU is the interaction ratio between the initial bounding box on the current video frame and the initial bounding box on the previous video frame.

[0144] The weight of the initial target box on the current video frame can be set to a fixed value, such as weight=1. This embodiment does not limit the value of weight.

[0145] In step S323, position weighting information is calculated based on the position information of the initial target box in the current video frame, the position information of the initial target box in the previous video frame, the weight information of the initial target box in the current video frame, and the weight information of the initial target box in the previous video frame.

[0146] In detail, the position weighting information of the final target bounding box of the sensitive target on the current video frame can be determined using the following formula.

[0147] pos=(pos·weight+pre_pos·pre_weight) / (weight+pre_weight)

[0148] Where pos is the position of the initial bounding box of the sensitive target in the current video frame, weight is the weight of the initial bounding box in the current video frame, pre_pos is the position of the initial bounding box in the previous video frame, and pre_weight is the weight of the initial bounding box in the previous video frame. In this embodiment of the disclosure, the weight of the initial bounding box in the current video frame can be set to a fixed value, for example, weight = 1.

[0149] In step S324, a new bounding box is established based on the position-weighted information, which serves as the final target bounding box for the sensitive target on the current video frame.

[0150] Through the above calculations, the position-weighted information of the target bounding box of the sensitive target in the current video frame can be obtained. Based on this position-weighted information, a new bounding box can be established, which is the final target bounding box of the sensitive target. By calculating the position-weighted information of all video frames in the image to be detected except the first frame, the final target bounding box of the same target in all video frames except the first frame is obtained by position-weighting the initial target bounding boxes of the two preceding and following frames. The final target bounding box of the sensitive target will not have inter-frame jitter, eliminating the inter-frame jitter phenomenon caused by errors in the quantization process, improving visual stability and viewing experience.

[0151] In some embodiments, the image target detection model processed by the above-described image target detection model processing method can, when used, quantize nodes suitable for quantization and complete model conversion and other steps. In actual tasks, the image to be detected can first undergo data conversion and quantization, and then be cached.

[0152] In this embodiment, the processes of preprocessing the image to be detected and calling the image target detection model for target detection can be executed in parallel. In this embodiment, the preprocessing and target detection processes can be performed simultaneously, making full use of multiple processors in the device to accelerate task processing. For example, the preprocessing of the image data to be detected can be run using chips such as GPU, CPU, or DSP, while the target detection process can be run using a different chip than the preprocessing process. Of course, the same chip can also be used for the preprocessing process. By executing the data preprocessing and target detection processes in parallel, multiple processors can be fully utilized to perform tasks, optimizing the terminal's task execution performance.

[0153] Understandably, before calling the image object detection model for object detection, the model can be converted to ONNX format according to the chip's operational requirements, and the opset version can be configured to match the version requirements in the chip's SDK. Additionally, the model's input size can be configured as needed, and appropriate parameters can be configured if model simplification is required.

[0154] It is understandable that after the image target detection model performs target detection on the image to be detected, it can also perform data dequantization and data transformation. Data dequantization and data transformation can be performed according to actual needs, and will not be elaborated here.

[0155] In some embodiments, after the image target detection model detects a sensitive target, the sensitive target can be desensitized. For example, if the sensitive target in an image or video is a face or license plate, the detected face and license plate can be blurred. This can prevent the dissemination of private information and protect personal privacy data.

[0156] like Figure 7 The diagram shown is an example of the application of an image target detection method on a vehicle-mounted platform, as provided in an exemplary embodiment of this disclosure.

[0157] On the vehicle infotainment platform, the in-vehicle camera application module can call the in-vehicle camera through the application programming interface (API) provided by the in-vehicle camera to start or stop the camera. The in-vehicle camera application module is configured with different application programs (APPs). The framework layer, through the API provided by the in-vehicle camera, and then through the hardware abstraction layer (HAL), performs the start or stop operation of the in-vehicle camera. The in-vehicle camera transmits captured video or image data to the camera API in the framework layer through the hardware abstraction layer (HAL).

[0158] Data annotation, model training, and optimization of the image object detection model can be performed through a server connected to the vehicle's infotainment platform. The image object detection model can be deployed in the framework layer of the vehicle's infotainment platform, including but not limited to model conversion and quantization. The quantization process here can involve quantizing nodes that are suitable for quantization after the above-mentioned discrimination process.

[0159] The Framework Layer performs target recognition on video or image data received through the camera API by calling the image target detection model and post-processing nodes that are not suitable for quantization. It then performs motion compensation on identified sensitive targets to eliminate inter-frame jitter and desensitizes the identified sensitive targets. Finally, the desensitized video or image data is transmitted to the vehicle camera application module for display on the app's page.

[0160] Through this processing flow, the image target detection model on the vehicle platform can run faster and more accurately without increasing the consumption of vehicle platform system resources and computing power, reducing errors in the video or image desensitization process, and improving the accuracy and speed of the desensitization task processing flow.

[0161] Based on the same inventive concept, such as Figure 8 The diagram shown is a block diagram of an image target detection model processing apparatus 700 provided in an exemplary embodiment of this disclosure. The image target detection model processing apparatus 700 includes a filtering module 701 and a processing module 702. Wherein:

[0162] The filtering module 701 is configured to determine quantizable nodes in the image target detection model, wherein the value range of the feature data output by the quantizable nodes is within the same value range.

[0163] The processing module 702 is configured to perform format conversion and node parameter quantization on the quantizable processing nodes to convert the format of the image target detection model into a chip-callable model format, wherein the node parameters of the quantizable processing nodes in the image target detection model after format conversion and quantization are integers.

[0164] Based on the same inventive concept, this disclosure also provides an image target detection device, including a detection module. Wherein:

[0165] The detection module is configured to call an image target detection model to perform target detection on the image to be detected, and obtain the sensitive targets in the image to be detected, wherein the image target detection model is processed by the above-described processing method.

[0166] Figure 9 This is a block diagram of an electronic device 800 provided in an exemplary embodiment of this disclosure. (See diagram below.) Figure 9 As shown, the electronic device 800 may include a processor 801 and a memory 802. The electronic device 800 may also include one or more of a multimedia component 803, an input / output (I / O) interface 804, and a communication component 805.

[0167] The processor 801 controls the overall operation of the electronic device 800 to complete all or part of the steps in the image target detection model processing method or image target detection method described above. The memory 802 stores various types of data to support the operation of the electronic device 800. This data may include, for example, instructions for any application or method operating on the electronic device 800, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 802 or transmitted via communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 805 is used for wired or wireless communication between the electronic device 800 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 805 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0168] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the processing method or image target detection method of the image target detection model described above.

[0169] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When executed by a processor, these program instructions implement the processing method of the image target detection model or the steps of the image target detection method described above. For example, the computer-readable storage medium may be the memory 802 including the program instructions described above, which may be executed by the processor 801 of the electronic device 800 to complete the processing method of the image target detection apparatus or the image target detection method described above.

[0170] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the processing method or image target detection method of the image target detection apparatus described above when executed by the programmable device.

[0171] This disclosure also provides a vehicle that includes the electronic equipment provided in this disclosure.

[0172] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0173] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0174] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A processing method for an image target detection model, characterized in that, include: In the image target detection model, quantizable nodes are identified, and the value range of the feature data output by the quantizable nodes is within the same value range. This includes: calling the image target detection model to perform target detection on the sample image, obtaining first output data after target detection at each node; quantizing the image target detection model; calling the quantized image target detection model to perform target detection on the sample image, obtaining second output data after target detection at each node; and determining quantizable nodes in the image target detection model based on the first output data and the second output data. The quantizable processing nodes are subjected to format conversion and node parameter quantization to convert the format of the image target detection model into a chip-callable model format. The image target detection model includes multiple intermediate layers. The step of calling the quantized image target detection model to perform target detection on the sample image and obtaining the second output data after target detection at each node includes: Each intermediate layer obtains feature data after performing target detection on the sample image; Determine whether the value range of the feature data corresponding to each intermediate layer is within the same value range; The second output data after all nodes have performed target detection is obtained from the intermediate layer where the value range of the feature data is within the same value range.

2. The processing method of the image target detection model according to claim 1, characterized in that, The step of determining the quantizable nodes in the image target detection model based on the first output data and the second output data includes: Determine the precision deviation between the second output data and the first output data, and confirm the node whose precision deviation is less than a preset deviation as the quantifiable processing node.

3. The processing method of the image target detection model according to claim 2, characterized in that, The method further includes: Nodes whose accuracy deviation is greater than or equal to a preset deviation are identified as unquantifiable processing nodes. The format of the unquantizable processing node is converted.

4. The processing method of the image target detection model according to claim 1, characterized in that, The steps to obtain the first output data after object detection at each node include: Each node obtains the first target coordinate value, first category probability, or first confidence level for the target detection in the sample image.

5. The processing method of the image target detection model according to claim 1, characterized in that, The steps for obtaining the second output data after target detection for all nodes in the intermediate layer where the value ranges of the feature data are within the same value range include: The value range of the feature data is obtained in the middle layer where the value range is the same, and each node obtains the second target coordinate value, second category probability, or second confidence level for the target detection in the sample image.

6. The processing method of the image target detection model according to claim 2, characterized in that, The steps of determining the precision deviation between the second output data and the first output data, and confirming that nodes with a precision deviation less than a preset deviation are quantifiable processing nodes, include: Nodes whose deviation between the first target coordinate value and the second target coordinate value is less than a preset coordinate value deviation are identified as quantifiable processing nodes; or Nodes whose deviation between the first category probability and the second category probability is less than a preset category probability deviation are identified as quantifiable processing nodes; or Nodes whose deviation between the first confidence level and the second confidence level is less than a preset confidence level deviation are identified as quantifiable nodes.

7. The processing method of the image target detection model according to claim 1, characterized in that, The steps for format conversion and node parameter quantization of the quantizable nodes include: Multiple pre-obtained feature images are used as input images for the quantization process, and the quantizable processing nodes are quantized. The multiple feature images include multiple brightness feature images, multiple scene feature images, and / or multiple illumination feature images.

8. An image target detection method, characterized in that, include: An image target detection model is invoked to perform target detection on the image to be detected, thereby obtaining sensitive targets in the image to be detected, wherein the image target detection model is processed by the processing method described in any one of claims 1 to 7.

9. The image target detection method according to claim 8, characterized in that, The steps of calling an image target detection model to perform target detection on the image to be detected and obtaining sensitive targets in the image include: Error and redundant borders in the bounding boxes obtained by target detection of the image to be detected are removed so that each sensitive target is selected by an initial target box; When the image to be detected includes multiple video frames, the final target bounding box of the sensitive target in the current video frame is determined based on the initial target bounding box on the current video frame and the initial target bounding box on the previous video frame.

10. The image target detection method according to claim 9, characterized in that, The steps for removing erroneous and redundant borders from the bounding boxes obtained by target detection of the image to be detected include: Calculate the confidence score and class probability of each bounding box obtained by target detection on the image to be detected; Among the bounding boxes with a confidence level greater than a preset confidence threshold, the bounding box with the highest category probability is used as the initial target box for the sensitive target.

11. The image target detection method according to claim 10, characterized in that, When the image to be detected comprises multiple video frames, the step of determining the final target bounding box of the sensitive target in the current video frame based on the initial target bounding box on the current video frame and the initial target bounding box on the previous video frame includes: Calculate the overlap area between the initial target bounding box on the current video frame and the initial target bounding box on the previous video frame; If the overlap area between the initial target box on the current video frame and the initial target box on the previous video frame is greater than a preset overlap threshold, calculate the weight information of the initial target box on the current video frame and the weight information of the initial target box on the previous video frame. Based on the position information of the initial target box in the current video frame, the position information of the initial target box in the previous video frame, the weight information of the initial target box in the current video frame, and the weight information of the initial target box in the previous video frame, position weight information is calculated; A new bounding box is established based on the location-weighted information, which serves as the final target bounding box for the sensitive target in the current video frame.

12. The image target detection method according to claim 8, characterized in that, The method includes: After obtaining the image to be detected, the image to be detected is converted and quantized, and the converted and quantized image to be detected is cached. The image target detection model is invoked to perform target detection on the cached image to be detected, thereby obtaining the sensitive targets in the image to be detected.

13. The image target detection method according to claim 12, characterized in that, After the step of calling the image target detection model to perform target detection on the image to be detected and obtaining the sensitive targets in the image to be detected, the method further includes: The sensitive targets are desensitized.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the processing method of the image target detection model according to any one of claims 1 to 7 or the image target detection method according to any one of claims 8 to 13.

15. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor is configured to execute the computer program in the memory to implement the processing method of the image target detection model according to any one of claims 1 to 7 or the image target detection method according to any one of claims 8 to 13.

Images