Methods, equipment, and media for detecting target visual elements in images.

Abstract

This disclosure provides a method, device, and medium for detecting target visual elements in an image, relating to the field of image processing technology. The method includes: acquiring an original image to be detected; obtaining a detection result output by the target visual element detection model based on the original image and a pre-trained target visual element detection model; the target visual element detection model includes a backbone network, a neck network, and a detection head connected in sequence; the backbone network includes at least a plurality of global context feature enhancement modules for aggregating global context information through a linear complexity attention mechanism to suppress noise features in the received image data; the backbone network is used to perform noise suppression on at least the original image to obtain multiple first feature maps of different scales; the neck network is used to perform feature fusion processing on the multiple first feature maps of different scales to obtain multiple second feature maps of different scales; and the detection head is used to generate detection results corresponding to each second feature map.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a method, device, and medium for detecting target visual elements in an image. Background Technology

[0002] With the rapid development of the broadcast television and online audiovisual industries, massive amounts of video data require real-time monitoring and review. Identifying target visual elements within video footage is crucial for ensuring safe broadcasting and industry oversight; these target visual elements can be specific visual objects that need to be identified or reviewed.

[0003] Currently, in low-light environments such as nighttime, directly identifying target visual elements in the video footage is typically very difficult. To reduce the difficulty of identifying target visual elements in low-light environments, a sequential architecture of image enhancement followed by target detection is usually employed. Specifically, a separate pre-image enhancement algorithm (such as the traditional Retinex algorithm or deep learning-based generative adversarial networks, low-light image enhancement networks, etc.) is first introduced to brighten, denoise, and enhance the contrast of the video footage. The processed video footage is then input into a general target detection model to identify target visual elements in the video. However, most existing pre-image enhancement algorithms are optimized for subjective human visual perception (such as pursuing smoothness and uniform brightness), aiming to generate a visually clearer intermediate image by repairing visual defects in the input image. Therefore, during the restoration process, the pre-image enhancement algorithm often misidentifies the image features corresponding to tiny target visual elements in the video image as noise and removes them along with the image features. This leads to the problem that the pre-image enhancement algorithm cannot identify tiny target visual elements in the video image. Summary of the Invention

[0004] One object of this disclosure is to provide a new technical solution for detecting target visual elements in an image.

[0005] According to a first aspect of this disclosure, a method for detecting target visual elements in an image is provided, the method comprising:

[0006] Obtain the original image to be detected;

[0007] Based on the original image and the pre-trained target visual element detection model, the detection result output by the target visual element detection model is obtained;

[0008] The detection result includes the location information and category of the target visual element in the original image. The target visual element detection model includes a backbone network, a neck network, and a detection head connected in sequence. The backbone network includes at least multiple global context feature enhancement modules. The global context feature enhancement modules are used to aggregate global context information through a linear complexity attention mechanism to suppress noise features in the received image data. The backbone network is used to perform noise suppression on at least the original image to obtain multiple first feature maps of different scales. The neck network is used to perform feature fusion processing on the multiple first feature maps of different scales to obtain multiple second feature maps of different scales. The detection head is used to generate detection results corresponding to each of the second feature maps.

[0009] Optionally, the backbone network further includes multiple dynamic receptive field attention capture modules, multiple global context feature enhancement modules, and multiple dynamic receptive field attention capture modules connected in series. The dynamic receptive field attention capture modules are used to simulate receptive fields of different scales to dynamically adapt to changes in the size of elements in the image.

[0010] Optionally, each global context feature enhancement module includes multiple global enhancement blocks, a bottleneck layer corresponding to each global enhancement block, and a connection layer. The multiple global enhancement blocks are connected in series. Each bottleneck layer receives the output of the corresponding global enhancement block. The connection layer receives the output of each bottleneck layer and the input of the global context feature enhancement module. The output of the connection layer is the output of the global context feature enhancement module. The activation function in each global enhancement block is the SiLU activation function, which is used to reduce the computational complexity to linear.

[0011] Optionally, each of the dynamic receptive field attention capture modules includes multiple parallel dilated convolution branches with different dilation rates, a first pointwise convolution parallel to the dilated convolution branches, a splicing layer that receives the output of each dilated convolution branch, a second pointwise convolution that receives the output of the splicing layer, and a summing module that sums the output of the second pointwise convolution with the output of the first pointwise convolution. The output of the summing module is used as the output of the dynamic receptive field attention capture module, and the input of the dilated convolution branches is used as the input of the dynamic receptive field attention capture module. Each dilated convolution branch includes a parameterless energy function attention mechanism block.

[0012] Optionally, the neck network includes a multi-scale visual-semantic adaptive fusion module, used to perform feature weighted fusion processing on multiple first feature maps of different scales to obtain multiple second feature maps of different scales.

[0013] Optionally, the pre-trained target visual element detection model is pre-trained in the following manner:

[0014] Obtain a training sample set, which includes multiple sets of training samples. Each training sample includes a sample image and a sample label corresponding to the sample image. The sample image includes target visual elements, and the sample label includes the category and location information of the target visual elements in the sample image.

[0015] Based on the training sample set and a preset boundary regression loss function based on geometric convex hull constraints, the initial target visual element detection model is trained to obtain a trained target visual element detection model. The boundary regression loss function based on geometric convex hull constraints includes a convex hull penalty term describing the area of ​​the gap outside the predicted and ground truth boxes within the area of ​​the minimum circumscribed convex hull that encloses the predicted and ground truth boxes, and / or a diagonal alignment auxiliary term describing the distance between the corner points on the diagonal of the predicted box and the corresponding corner points on the diagonal of the ground truth box. The predicted box is the position description box corresponding to the position information of the target visual element in the sample image detected by the initial target visual element detection model, and the ground truth box is the position description box corresponding to the position information of the target visual element in the sample label.

[0016] Optionally, the boundary regression loss function based on geometric convex hull constraints is:

[0017] ;

[0018] in, This is the loss value. The area is the intersection of the ground truth bounding box and the predicted bounding box. To adjust the penalty coefficient for the weights, Let be the area of ​​the minimum outer convex hull that encloses both the predicted bounding box and the ground truth bounding box. The area is the union of the ground truth bounding box and the predicted bounding box. The Euclidean distance between the preset top corner of the predicted bounding box and the preset top corner of the ground truth bounding box is given. The preset lower corner point is the Euclidean distance between the preset lower corner point of the predicted bounding box and the preset lower corner point of the ground truth bounding box. The preset upper corner point and the preset lower corner point are corner points on the diagonal. The width of the minimum circumscribed convex hull. The height of the minimum circumscribed convex hull is given.

[0019] Optionally, the method further includes:

[0020] The location information and category of the target visual elements in the original image are mapped to the original image to generate a labeled image with a bounding box;

[0021] And / or, if the detection result indicates that the original image includes the target visual element and is an image in the video to be reviewed, the timecode of the original image is recorded.

[0022] According to a second aspect of this disclosure, an electronic device is provided, the electronic device including a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to invoke the computer instructions from the memory to perform the method as described in any one of the first aspects.

[0023] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the first aspects.

[0024] This disclosure provides a method for detecting target visual elements in an image. The method includes: acquiring an original image to be detected; obtaining a detection result output by the target visual element detection model based on the original image and a pre-trained target visual element detection model; wherein the detection result includes the location information and category of the target visual elements in the original image; the target visual element detection model includes a backbone network, a neck network, and a detection head connected sequentially; the backbone network includes at least multiple global context feature enhancement modules, which aggregate global context information through a linear complexity attention mechanism to suppress noise features in the received image data; the backbone network performs noise suppression on at least the original image to obtain multiple first feature maps of different scales; the neck network performs feature fusion processing on the multiple first feature maps of different scales to obtain multiple second feature maps of different scales; and the detection head generates detection results corresponding to each second feature map. This method eliminates the need for computationally intensive pre-image enhancement algorithms as in traditional techniques, blocking noise propagation at the initial stage of feature encoding and preventing the texture details of the target visual elements in the original image from being submerged by noise during downsampling. Thus, even tiny target visual elements are not submerged by noise. Therefore, this method can improve the recognition rate of visual elements of small targets.

[0025] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0027] Figure 1 This is a flowchart illustrating a method for detecting target visual elements in an image, as provided in this disclosure.

[0028] Figure 2 This is a schematic diagram of the structure of a target visual element detection model provided in this disclosure;

[0029] Figure 3 This is a schematic diagram of the structure of a global context feature enhancement module and a global enhancement block provided in this disclosure;

[0030] Figure 4 This is a schematic diagram of the structure of a dynamic receptive field attention capture module provided in this disclosure;

[0031] Figure 5 This is a schematic diagram of the structure of a device for detecting target visual elements in an image, as provided in this disclosure;

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this disclosure. Detailed Implementation

[0033] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0034] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0035] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0036] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0037] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0038] It should be noted that all actions involving the collection, storage, use, processing, transmission, provision, publication, and deletion of data in this disclosure are carried out in accordance with local data protection regulations and with the consent of the relevant data owners.

[0039] This disclosure provides a method for detecting target visual elements in an image, such as... Figure 1 As shown, the method includes the following steps S1100 and S1200.

[0040] Step S1100: Obtain the original image to be detected.

[0041] In this embodiment of the disclosure, the original image to be detected is the image for which target visual elements are to be detected. The target visual elements are, for example, text or images related to illegal or prohibited activities, negative temptations, infringement of rights, or protection of minors.

[0042] The sources of the original images include: real-time broadcast television signal streams, online video-on-demand files, and high-definition archived materials. When the original image is an encoded video stream, the original image can be specifically obtained by: performing basic decoding operations on the encoded video stream to obtain a decoded video stream; extracting keyframe images from the decoded video stream according to a preset sampling frequency, and using these keyframe images as the original image.

[0043] Step S1200: Based on the original image and the pre-trained target visual element detection model, obtain the detection result output by the target visual element detection model.

[0044] The detection results include the location information and category of the target visual elements in the original image. The target visual element detection model includes a backbone network, a neck network, and a detection head connected in sequence. The backbone network includes at least multiple Global Context Feature Enhancement (GCFE) modules. The GCFE modules are used to aggregate global context information through a linear complexity attention mechanism to suppress noise features in the received image data, thereby achieving feature enhancement. The backbone network is used to suppress noise in the original image to obtain multiple first feature maps of different scales. The neck network is used to perform feature fusion processing on the multiple first feature maps of different scales to obtain multiple second feature maps of different scales. The detection head is used to generate detection results corresponding to each second feature map.

[0045] In one embodiment of this disclosure, the original image is first converted into image data conforming to the data matrix format input to a pre-trained target visual element detection model. Further, the image data is input into the pre-trained target visual element detection model, which outputs the location information and category detection results for the target visual elements in the original image. It is understood that if the original image does not contain target visual elements, the detection result is empty.

[0046] In this embodiment of the disclosure, an improvement is made to the traditional YOLO model, providing a new structural target visual element detection model. For example... Figure 2 As shown, the target visual element detection model includes a backbone network, a neck network, and a detection head connected in sequence. The backbone network does not use the standard convolutional modules of the traditional YOLO model, but instead includes at least several global context feature enhancement modules. These global context feature enhancement modules are specifically used to aggregate global context information through a linear complexity attention mechanism without increasing GPU memory computation, enabling accurate identification and suppression of noise features (such as random physical high-frequency shot noise generated by the photosensitive element) in the received image data. This eliminates the need for computationally intensive pre-image enhancement algorithms as in traditional techniques. Instead, it blocks noise propagation at the initial stage of feature encoding, preventing the texture details of the target visual elements in the original image from being submerged by noise during downsampling. Thus, even tiny target visual elements are not obscured by noise. Therefore, the target visual element detection method provided in this disclosure can improve the recognition rate of tiny target visual elements. Based on this, after receiving image data, the backbone network performs noise suppression on the original image to obtain multiple first feature maps of different scales, which are then output to the neck network.

[0047] The neck network is used to perform feature fusion processing on first feature maps of different scales to obtain multiple second feature maps of different scales, and outputs them to the detection head. The detection head is used to generate detection results corresponding to each second feature map. In one embodiment of this disclosure, the neck network and the detection head can adopt the corresponding structure in the traditional YOLO model.

[0048] It should be noted that, Figure 2 The example shown uses Head1, Head2, Head3 and the fusion layer dense as detection heads.

[0049] Based on the above, this disclosure provides a method for detecting target visual elements in an image. The method includes: acquiring an original image to be detected; obtaining a detection result output by the target visual element detection model based on the original image and a pre-trained target visual element detection model; wherein the detection result includes the location information and category of the target visual elements in the original image; the target visual element detection model includes a backbone network, a neck network, and a detection head connected sequentially; the backbone network includes at least multiple global context feature enhancement modules, which aggregate global context information through a linear complexity attention mechanism to suppress noise features in the received image data; the backbone network performs noise suppression on at least the original image to obtain multiple first feature maps of different scales; the neck network performs feature fusion processing on the multiple first feature maps of different scales to obtain multiple second feature maps of different scales; and the detection head generates detection results corresponding to each second feature map. This method eliminates the need for computationally intensive pre-image enhancement algorithms as in traditional techniques, blocking noise propagation at the initial stage of feature encoding and preventing the texture details of the target visual elements in the original image from being submerged by noise during downsampling. Thus, even tiny target visual elements will not be submerged by noise. Therefore, this method can improve the recognition rate of visual elements of small targets.

[0050] In one embodiment of this disclosure, the backbone network further includes multiple dynamic receptive field attention capture modules, multiple global context feature enhancement modules, and multiple dynamic receptive field attention capture modules (DRAM) connected in series alternately. The dynamic receptive field attention capture modules are used to simulate receptive fields of different scales to dynamically adapt to changes in the size of elements in the image.

[0051] Because target visual elements in the original image are often extremely small in size and have variable shapes—for example, a flag fluttering in the wind in the distance, or a logo in a corner of the original image—traditional YOLO models use fixed convolutional kernels for feature extraction. This results in the traditional YOLO backbone network being unable to effectively cover these extremely small and variable-shaped target visual elements in the original image. To address this, in this embodiment, a dynamic receptive field attention capture module is introduced into the backbone network. This module simulates receptive fields of different scales to dynamically adapt to changes in the size of elements in the image, thereby achieving adaptive focusing on minute image elements. This avoids the problem of the traditional YOLO model's fixed convolutional kernel and fixed receptive field, which prevents it from effectively capturing the features of extremely small and variable-shaped target visual elements in the original image. Thus, the pre-trained target visual element detection model can improve the accuracy of detecting extremely small and variable-shaped target visual elements in the original image. In other words, by introducing a dynamic receptive field attention capture module into the backbone network, the target visual element detection method in the image provided in this disclosure can further improve the recognition rate of small target visual elements.

[0052] In one embodiment of this disclosure, the backbone network is as follows: Figure 2 As shown, the backbone network includes GCFE1, DRAM1, GCFE2, DRAM2, GCFE3, DRAM3, and GCFE4 connected sequentially. GCFE1 initially performs noise suppression on the received image data, obtaining a third feature map; DRAM1 extracts features from the third feature map using a small receptive field, obtaining a fourth feature map; GCFE2 further performs noise suppression on the fourth feature map, obtaining a fifth feature map; DRAM2 extracts features from the fifth feature map using a medium receptive field, obtaining a sixth feature map; GCFE3 further performs noise suppression on the sixth feature map, obtaining a seventh feature map; DRAM3 extracts features from the seventh feature map using a large receptive field, obtaining an eighth feature map; and GCFE4 further performs noise suppression on the eighth feature map, obtaining a ninth feature map. Thus, the backbone network can accurately extract features from small, medium, and large receptive fields in the original image. Of course, more sets of GCFE and DRAM can be added to the backbone network to achieve accurate extraction of features from more levels of receptive fields; this disclosure does not limit this.

[0053] In one embodiment of this disclosure, such as Figure 3As shown, each global context feature enhancement module includes multiple global enhancement blocks (GEBs), a bottleneck layer corresponding to each GEB, and a connection layer. The multiple GEBs are connected in series. Each bottleneck layer receives the output of the corresponding GEB. The connection layer receives the output of each bottleneck layer and the input of the global context feature enhancement module. The output of the connection layer is the output of the global context feature enhancement module. The activation function in each GEB is the SiLU activation function, which is used to reduce the computational complexity to linear.

[0054] It should be noted that, Figure 3 The example shown uses global enhancement blocks GEB1, GEB2, and GEB3.

[0055] In this embodiment, taking the global enhancement block GEB3 as an example, GEB3 is the core of the global context feature enhancement module for implementing global context aggregation, noise identification, and suppression, and internally includes a linear attention complexity attention mechanism. The structure of GEB3 can be as follows: Figure 3 As shown. Based on Figure 3 The structure shown, GEB1 is specifically used for: splitting the input feature map into three feature branches: Q, K, and V; preparing for global attention calculation by performing linear mapping on the three feature branches Q (Quary), K (Key), and V (Value) respectively, regularizing the channel dimensions to adapt to attention calculation; replacing the traditional Softmax with the SiLU activation function, and reducing the complexity of the corresponding linear mapping results obtained by performing linear mapping on the Q and K feature branches respectively, so as to reduce the attention calculation complexity from quadratic O(N²) to linear O(N) (N is the number of pixels in the feature map), and then multiplying them to realize global context similarity calculation and generate the weights of the global context vector; multiplying the weights of the global context vector with the corresponding linear mapping results obtained by performing linear mapping on the V feature branch, and then using the Scale module to perform lightweight numerical normalization, so as to realize global noise reduction by suppressing noise response weights through attention weighting, while preserving and strengthening the structural semantic features of the target visual elements. The use of the Scale module for lightweight numerical normalization effectively prevents gradient explosion and ensures stable end-to-end convergence without introducing complex statistical calculations of the traditional LayerNorm.

[0056] The bottleneck layer receives the output of the corresponding global enhancement block and uses it to perform local feature optimization on the globally denoised feature map obtained from the global enhancement block, resulting in a locally optimized feature map. And, Figure 3The example shown uses Bottleneck1, Bottleneck2, and Bottleneck3 as bottleneck layers.

[0057] The connection layer receives the locally optimized feature map from each bottleneck layer and the feature map input to the global context feature enhancement module, and performs feature aggregation and fusion to obtain the global denoising feature map output by the global context feature enhancement module. Figure 3 The example shown uses Connect as the connection layer.

[0058] Furthermore, since the activation function in the global enhancement block is the SiLU activation function, which reduces the computational complexity to linear, the global context feature enhancement module can achieve global noise reduction with low computational consumption. This facilitates the end-to-end lightweight deployment of the target visual element detection method in the image provided in this disclosure.

[0059] In one embodiment of this disclosure, such as Figure 4 As shown, each dynamic receptive field attention capture module includes multiple parallel dilated convolution branches with different dilation rates, a first pointwise convolution parallel to the dilated convolution branches, a splicing layer that receives the output of each dilated convolution branch, a second pointwise convolution that receives the output of the splicing layer, and a summing module that sums the output of the second pointwise convolution with the output of the first pointwise convolution. The output of the summing module is used as the output of the dynamic receptive field attention capture module, and the input of the dilated convolution branches is used as the input of the dynamic receptive field attention capture module. Each dilated convolution branch includes a parameterless energy function attention mechanism block.

[0060] In this embodiment, dilated convolution branches with different dilation rates are used in parallel as the core feature extraction units of the dynamic receptive field attention capture module.

[0061] It should be noted that, Figure 4 The example shown is a three-way parallel dilated convolution branch with different dilation rates.

[0062] For dilated convolution branch 1, it is a "small receptive field branch" in the multi-path parallel dilated convolution branch, which includes 1×1DWConv that performs 1×1 dilated convolution, 3×3DWConv that performs 3×3 dilated convolution with an expansion rate of 1, and SimAM1, which is a parameterless energy function attention mechanism block, connected in sequence.

[0063] For dilated convolution branch 2, it is the "middle receptive field branch" in the multi-path parallel dilated convolution branch, which includes 3×3DWConv that performs 3×3 dilated convolution, 3×3DWConv that performs 3×3 dilated convolution with an expansion rate of 3, and SimAM2, which is a parameterless energy function attention mechanism block, connected in sequence.

[0064] For dilated convolution branch 3, it is the "large receptive field branch" in the multi-path parallel dilated convolution branch, which includes 5×5DWConv that performs 5×5 dilated convolution, 3×3DWConv that performs 3×3 dilated convolution with an expansion rate of 5, and the third SimAM3, which is a parameterless energy function attention mechanism block.

[0065] Taking dilated convolution branch 1 as an example, the 1×1DWConv is used to perform preliminary local spatial feature extraction and lightweight compression without changing the size of the input feature map, providing feature preprocessing for the subsequent 3×3DWConv with a dilation rate of 1. The 3×3DWConv with a dilation rate of 1 is used to extract fine local spatial features, capturing details such as edges and textures of small targets (such as logos and flags), while maintaining lightweight computation. SimAM1 is used to automatically infer the linear separability of the target region and the background region based on the energy difference of neurons, adaptively enhancing the response weights to effective features such as flag edges and logo outlines, suppressing invalid interference in dark backgrounds, and ensuring that the network can accurately lock onto faint targets. The descriptions of dilated convolution branches 2 and 3 are similar to those of dilated convolution branch 1 and will not be repeated here.

[0066] The first pointwise convolution is used to extract channel semantic features in a lightweight manner, serving as a baseline branch to compensate for the lack of channel features in dilated convolution.

[0067] The concatenation layer is used to concatenate the output features of the three dilated convolution branches along the channel dimension, achieving multi-scale feature aggregation with multiple dilation rates and receptive fields, integrating scattered local / global spatial features into a complete multi-scale feature map. It should be noted that... Figure 4 The example shown uses Concat4 as the splicing layer.

[0068] The second pointwise convolution is used to perform 1×1 pointwise convolution on the multi-scale feature map output by the splicing layer. Through linear transformation between channels, the aggregated multi-scale spatial features are fused with the channel semantic features.

[0069] The summation module is used to add and fuse the feature map output by the second pointwise convolution with the feature map output by the first pointwise convolution. This can further enhance the response values ​​of the effective feature regions in the second pointwise convolution output feature map and the first pointwise convolution output feature map, while the response values ​​of the background interference regions cancel each other out, thereby achieving effective feature enhancement and background feature suppression.

[0070] The dynamic receptive field attention capture module provided in this embodiment dynamically adjusts the receptive field range by setting dilated convolution branches with different dilation rates in parallel, and adaptively enhances the contrast between the edge of the small target and the dark background by combining a parameterless energy function attention mechanism, ensuring that the feature extraction network can clearly "see" the outline of the small target.

[0071] In one embodiment of this disclosure, in order to solve the problem of semantic fragmentation in images captured by lens zooming, such as... Figure 2 As shown, the neck network includes a multi-scale visual semantic adaptive fusion module (MSAF), which performs feature weighted fusion processing on multiple first feature maps of different scales to obtain multiple second feature maps of different scales.

[0072] In this embodiment of the disclosure, the neck network is based on the original splicing layer (such as... Figure 2 Based on the structure consisting of Concat1, Concat2, and Concat3 in the backbone network, a multi-scale visual semantic adaptive fusion module (MSAF) is added. The multi-scale visual semantic adaptive fusion module performs weighted fusion of features from different levels of the backbone network (including deep high-level semantic features and shallow detailed texture features). Figure 2 The example demonstrates the use of MSAF1, MSAF2, and MSAF3 as examples of multi-scale visual-semantic adaptive fusion modules. Figure 2 Taking the target visual element detection model structure shown as an example, the fusion method of the multi-scale visual semantic adaptive fusion module can be expressed by the following formula.

[0073] (Formula 1)

[0074] in, ,and This is the output of the multi-scale visual-semantic adaptive fusion module. The deep semantic feature map obtained by splicing the splicing layers, The shallow detail feature map obtained by stitching together the stitched layers. This is the transition feature map of the middle layer obtained by splicing the splicing layers.

[0075] Using Formula 1 above, the target visual element detection model can automatically learn which scale of features should be assigned higher weights at different spatial locations. For example, it emphasizes deep semantics when detecting large nearby banners, while emphasizing shallow details when detecting small icons in the distance. This mechanism effectively compensates for the semantic gaps in the feature pyramid when processing multi-source heterogeneous materials, ensuring the model's robustness to target visual elements of various sizes.

[0076] In one embodiment of this disclosure, the pre-trained target visual element detection model in step S1200 is implemented based on steps S1210 to S1220.

[0077] Step S1210: Obtain the training sample set.

[0078] The training sample set includes multiple training samples. Each training sample includes a sample image and a corresponding sample label. The sample image includes target visual elements, and the sample label includes the category and location information of the target visual elements in the corresponding sample image.

[0079] In this embodiment, to cover special scenarios such as breaking news at night, undercover investigations, and old historical footage, sample collectors can collect sample images containing various target visual elements (such as specific flags, station logos, illegal banners, and text labels). Considering physical damage in practical applications, sample images with low illumination, high noise, and extremely small target sizes (e.g., occupying less than 1% of the image) can be prioritized, and these target visual elements can be labeled with high-precision rectangular boxes. To simulate the input environment of real edge nodes and enhance the robustness of the model, some clear sample images also need to be manually processed with Gaussian noise or motion blur. In this way, a dedicated training sample set containing various lighting conditions and noise levels can be constructed.

[0080] Step S1220: Based on the training sample set and the preset boundary regression loss function based on geometric convex hull constraints, train the initial detection model of the target visual elements to be trained to obtain the trained target visual element detection model.

[0081] Based on the training sample set and the preset boundary regression loss function based on geometric convex hull constraints, the initial target visual element detection model is trained to obtain the trained target visual element detection model. The boundary regression loss function based on geometric convex hull constraints includes a convex hull penalty term describing the area of ​​the gap outside the predicted box and the ground truth box in the area of ​​the minimum outer convex hull that encloses the predicted box and the ground truth box, and / or a diagonal alignment auxiliary term describing the distance between the corner points on the diagonal of the predicted box and the corresponding corner points on the diagonal of the ground truth box. The predicted box is the position description box corresponding to the position information of the target visual element in the sample image detected by the initial target visual element detection model, and the ground truth box is the position description box corresponding to the position information of the target visual element in the sample label.

[0082] In this embodiment, the initial target visual element detection model to be trained has default parameter values ​​for each neuron in its initial state. Using a training sample set and a preset boundary regression loss function based on geometric convex hull constraints, the initial target visual element detection model is trained according to the traditional neural network training process to update the parameter values ​​of each neuron in the initial target visual element detection model until the updated initial target visual element detection model meets a preset convergence condition. The convergence condition can be exemplarily defined as the loss value corresponding to the boundary regression loss function based on geometric convex hull constraints being less than or equal to a preset loss value, or the initial target visual element detection model reaching a preset number of iterations. The preset loss value and the preset number of iterations can be set empirically.

[0083] It should be noted that, in this embodiment, traditional loss functions such as IoU are abandoned, and a boundary regression loss function based on geometric convex hull constraints is provided as a new loss function. This preset boundary regression loss function based on geometric convex hull constraints introduces a convex hull penalty term as a geometric constraint term. The convex hull penalty term describes the area of ​​the gaps excluding the predicted and ground truth boxes within the area of ​​the minimum bounding convex hull that encloses the predicted and ground truth boxes. The minimum bounding convex hull can be exemplarily represented as follows: Figure 2 As shown in the dashed box in the upper right corner. In this way, even if the intersection of the predicted box and the ground truth box is 0, the boundary regression loss function based on the geometric convex hull constraint can still generate a strong gradient signal pointing to the geometric center of the ground truth box through the area of ​​this gap, thereby achieving fast "long-distance pull-back".

[0084] Furthermore, the pre-defined boundary regression loss function based on geometric convex hull constraints introduces a diagonal alignment auxiliary term. This term describes the distances between the corner points on the diagonal of the predicted bounding box and their corresponding corner points on the diagonal of the ground truth bounding box. The distances between the corner points on the diagonal of the predicted bounding box and their corresponding corner points on the diagonal of the ground truth bounding box are: the Euclidean distance between the top-left corner of the predicted bounding box and the top-left corner of the ground truth bounding box, and the Euclidean distance between the bottom-right corner of the predicted bounding box and the bottom-right corner of the ground truth bounding box. Alternatively, the distances between the corner points on the diagonal of the predicted bounding box and their corresponding corner points on the diagonal of the ground truth bounding box are: the Euclidean distance between the top-right corner of the predicted bounding box and the top-right corner of the ground truth bounding box, and the Euclidean distance between the bottom-left corner of the predicted bounding box and the bottom-left corner of the ground truth bounding box.

[0085] The initial target visual element detection model can use the diagonal alignment auxiliary term to determine the optimization target as forcing the four corner points of the predicted box to accurately fit the four corner points of the real box, rather than determining it to make the predicted box close to the real box. In this way, the target visual element detection model trained can accurately detect the position information of easily deformable target visual elements such as flags.

[0086] In one embodiment of this disclosure, the boundary regression loss function based on geometric convex hull constraints is shown in Formula 2 below.

[0087] (Formula 2)

[0088] in, This is the loss value. This is the area of ​​the intersection of the ground truth bounding box and the predicted bounding box. To adjust the penalty coefficient for the weights, Let the area of ​​the minimum bounding convex hull that encloses both the predicted bounding box and the ground truth bounding box be [area]. This is the area of ​​the union of the ground truth bounding box and the predicted bounding box. The Euclidean distance between the preset top corner of the predicted bounding box and the preset top corner of the ground truth bounding box. The preset bottom corner of the predicted bounding box is the Euclidean distance between the preset bottom corner of the actual bounding box and the preset top corner of the predicted bounding box. The preset top corner and preset bottom corner are corners on the diagonal. The width of the minimum circumscribed convex hull. The height of the minimum circumscribed convex hull.

[0089] In this embodiment of the disclosure, when the preset top corner is the top left corner, the preset bottom corner is the bottom right corner. When the preset top corner is the top right corner, the preset bottom corner is the bottom left corner.

[0090] In this embodiment, For convex hull penalty terms, This is in response to the diagonal alignment guide. It should be noted that the diagonal alignment guide includes... Convex hull size via minimum circumscribed convex hull Normalization.

[0091] In one embodiment of this disclosure, the method for detecting target visual elements in an image provided by this disclosure further includes the following steps S1300 and / or S1400.

[0092] Step S1300: Map the position information and category of the target visual element in the original image to the original image to generate a marked image with a bounding box.

[0093] By using the above step S1300, when viewing the original image, the location and category of the target visual elements in the original image can be directly viewed.

[0094] Step S1400: If the detection result indicates that the original image includes the target visual element and is an image in the video to be reviewed, record the timecode of the original image.

[0095] By using the above step S1400, the location of the target visual element in the video to be reviewed can be determined.

[0096] In one embodiment of this disclosure, structured logs can be further generated according to the above steps S1300 and S1400, and the structured logs can be pushed to the management platform for subsequent manual review or automatic blocking processing, etc.

[0097] like Figure 5 As shown, this disclosure also provides a target visual element detection device 500 in an image, the target visual element detection device 500 comprising:

[0098] The acquisition module 510 is used to acquire the original image to be detected;

[0099] The detection module 520 is used to obtain the detection result output by the target visual element detection model based on the original image and the pre-trained target visual element detection model.

[0100] The detection result includes the location information and category of the target visual element in the original image. The target visual element detection model includes a backbone network, a neck network, and a detection head connected in sequence. The backbone network includes at least multiple global context feature enhancement modules. The global context feature enhancement modules are used to aggregate global context information through a linear complexity attention mechanism to suppress noise features in the received image data. The backbone network is used to perform noise suppression on at least the original image to obtain multiple first feature maps of different scales. The neck network is used to perform feature fusion processing on the multiple first feature maps of different scales to obtain multiple second feature maps of different scales. The detection head is used to generate detection results corresponding to each of the second feature maps.

[0101] It should be noted that the acquisition module 510 and the detection module 520 can be deployed in different devices.

[0102] In one embodiment of this disclosure, the backbone network further includes multiple dynamic receptive field attention capture modules, multiple global context feature enhancement modules, and multiple dynamic receptive field attention capture modules connected in series alternately. The dynamic receptive field attention capture modules are used to simulate receptive fields of different scales to dynamically adapt to changes in the size of elements in the image.

[0103] In one embodiment of this disclosure, each global context feature enhancement module includes a plurality of global enhancement blocks, a bottleneck layer corresponding to each global enhancement block, and a connection layer. The plurality of global enhancement blocks are connected in series. Each bottleneck layer receives the output of the corresponding global enhancement block. The connection layer receives the output of each bottleneck layer and the input of the global context feature enhancement module. The output of the connection layer is the output of the global context feature enhancement module. The activation function in each global enhancement block is the SiLU activation function, which is used to reduce the computational complexity to linear.

[0104] In one embodiment of this disclosure, each of the dynamic receptive field attention capture modules includes multiple parallel dilated convolution branches with different dilation rates, a first pointwise convolution parallel to the dilated convolution branches, a splicing layer that receives the output of each dilated convolution branch, a second pointwise convolution that receives the output of the splicing layer, and a summing module that sums the output of the second pointwise convolution with the output of the first pointwise convolution. The output of the summing module is used as the output of the dynamic receptive field attention capture module, and the input of the dilated convolution branches is used as the input of the dynamic receptive field attention capture module. Each dilated convolution branch includes a parameterless energy function attention mechanism block.

[0105] In one embodiment of this disclosure, the neck network includes a multi-scale visual-semantic adaptive fusion module, which performs feature weighted fusion processing on multiple first feature maps of different scales to obtain multiple second feature maps of different scales.

[0106] In one embodiment of this disclosure, the pre-trained target visual element detection model is pre-trained based on the following method:

[0107] Obtain a training sample set, which includes multiple sets of training samples. Each training sample includes a sample image and a sample label corresponding to the sample image. The sample image includes target visual elements, and the sample label includes the category and location information of the target visual elements in the sample image.

[0108] Based on the training sample set and a preset boundary regression loss function based on geometric convex hull constraints, the initial target visual element detection model is trained to obtain a trained target visual element detection model. The boundary regression loss function based on geometric convex hull constraints includes a convex hull penalty term describing the area of ​​the gap outside the predicted and ground truth boxes within the area of ​​the minimum circumscribed convex hull that encloses the predicted and ground truth boxes, and / or a diagonal alignment auxiliary term describing the distance between the corner points on the diagonal of the predicted box and the corresponding corner points on the diagonal of the ground truth box. The predicted box is the position description box corresponding to the position information of the target visual element in the sample image detected by the initial target visual element detection model, and the ground truth box is the position description box corresponding to the position information of the target visual element in the sample label.

[0109] In one embodiment of this disclosure, the boundary regression loss function based on geometric convex hull constraints is:

[0110] ;

[0111] in, This is the loss value. The area is the intersection of the ground truth bounding box and the predicted bounding box. To adjust the penalty coefficient for the weights, Let be the area of ​​the minimum outer convex hull that encloses both the predicted bounding box and the ground truth bounding box. The area is the union of the ground truth bounding box and the predicted bounding box. The Euclidean distance between the preset top corner of the predicted bounding box and the preset top corner of the ground truth bounding box is given. The preset lower corner point is the Euclidean distance between the preset lower corner point of the predicted bounding box and the preset lower corner point of the ground truth bounding box. The preset upper corner point and the preset lower corner point are corner points on the diagonal. The width of the minimum circumscribed convex hull. The height of the minimum circumscribed convex hull is given.

[0112] In one embodiment of this disclosure, the target visual element detection device 500 in an image includes:

[0113] The generation module is used to map the position information and category of the target visual elements in the original image to the original image to generate a marked image with a bounding box;

[0114] A recording module is used to record the timecode of the original image when the detection result indicates that the original image includes target visual elements and is an image in the video to be reviewed.

[0115] It should be noted that the acquisition module 510, detection module 520, generation module, and recording module can be deployed in different devices. This disclosure does not limit the deployment method of the acquisition module 510, detection module 520, generation module, and recording module.

[0116] like Figure 6 As shown, this disclosure also provides an electronic device 600, which includes a memory 610 and a processor 620. The memory 610 is used to store computer instructions, and the processor 620 is used to call the computer instructions from the memory 610 to perform the method as described in any of the above method embodiments.

[0117] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the above-described method embodiments.

[0118] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0119] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0120] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0121] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0122] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object programs written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., "C" or similar languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, can execute the computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.

[0123] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0124] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0125] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0127] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for detecting target visual elements in an image, characterized in that, The method includes: Obtain the original image to be detected; Based on the original image and the pre-trained target visual element detection model, the detection result output by the target visual element detection model is obtained; The detection result includes the location information and category of the target visual element in the original image. The target visual element detection model includes a backbone network, a neck network, and a detection head connected in sequence. The backbone network includes at least multiple global context feature enhancement modules. The global context feature enhancement modules are used to aggregate global context information through a linear complexity attention mechanism to suppress noise features in the received image data. The backbone network is used to perform noise suppression on at least the original image to obtain multiple first feature maps of different scales. The neck network is used to perform feature fusion processing on the multiple first feature maps of different scales to obtain multiple second feature maps of different scales. The detection head is used to generate detection results corresponding to each of the second feature maps. Each global context feature enhancement module includes multiple global enhancement blocks, a bottleneck layer corresponding to each global enhancement block, and a connection layer. The multiple global enhancement blocks are connected in series. Each bottleneck layer receives the output of the corresponding global enhancement block. The connection layer receives the output of each bottleneck layer and the input of the global context feature enhancement module. The output of the connection layer is the output of the global context feature enhancement module. The activation function in each global enhancement block is the SiLU activation function, which is used to reduce the computational complexity to linear.

2. The method according to claim 1, characterized in that, The backbone network also includes multiple dynamic receptive field attention capture modules, multiple global context feature enhancement modules and multiple dynamic receptive field attention capture modules are alternately connected in series, and the dynamic receptive field attention capture modules are used to simulate receptive fields of different scales to dynamically adapt to changes in the size of elements in the image.

3. The method according to claim 2, characterized in that, Each of the dynamic receptive field attention capture modules includes multiple parallel dilated convolution branches with different dilation rates, a first pointwise convolution parallel to the dilated convolution branches, a splicing layer that receives the output of each dilated convolution branch, a second pointwise convolution that receives the output of the splicing layer, and a summing module that sums the output of the second pointwise convolution with the output of the first pointwise convolution. The output of the summing module is used as the output of the dynamic receptive field attention capture module, and the input of the dilated convolution branches is used as the input of the dynamic receptive field attention capture module. Each dilated convolution branch includes a parameterless energy function attention mechanism block.

4. The method according to claim 1, characterized in that, The neck network includes a multi-scale visual-semantic adaptive fusion module, which performs feature weighted fusion processing on multiple first feature maps of different scales to obtain multiple second feature maps of different scales.

5. The method according to claim 1, characterized in that, The pre-trained target visual element detection model is pre-trained based on the following method: Obtain a training sample set, which includes multiple sets of training samples. Each training sample includes a sample image and a sample label corresponding to the sample image. The sample image includes target visual elements, and the sample label includes the category and location information of the target visual elements in the sample image. Based on the training sample set and a preset boundary regression loss function based on geometric convex hull constraints, the initial target visual element detection model is trained to obtain a trained target visual element detection model. The boundary regression loss function based on geometric convex hull constraints includes a convex hull penalty term describing the area of ​​the gap outside the predicted and ground truth boxes within the area of ​​the minimum circumscribed convex hull that encloses the predicted and ground truth boxes, and / or a diagonal alignment auxiliary term describing the distance between the corner points on the diagonal of the predicted box and the corresponding corner points on the diagonal of the ground truth box. The predicted box is the position description box corresponding to the position information of the target visual element in the sample image detected by the initial target visual element detection model, and the ground truth box is the position description box corresponding to the position information of the target visual element in the sample label.

6. The method according to claim 5, characterized in that, The boundary regression loss function based on geometric convex hull constraints is: ； in, The loss value. The area is the intersection of the ground truth bounding box and the predicted bounding box. To adjust the penalty coefficient for the weights, Let be the area of ​​the minimum outer convex hull that encloses both the predicted bounding box and the ground truth bounding box. The area is the union of the ground truth bounding box and the predicted bounding box. The Euclidean distance between the preset top corner of the predicted bounding box and the preset top corner of the ground truth bounding box is given. The preset lower corner point is the Euclidean distance between the preset lower corner point of the predicted bounding box and the preset lower corner point of the ground truth bounding box. The preset upper corner point and the preset lower corner point are corner points on the diagonal. The width of the minimum circumscribed convex hull. The height of the minimum circumscribed convex hull is given.

7. The method according to claim 1, characterized in that, The method further includes: The location information and category of the target visual elements in the original image are mapped to the original image to generate a labeled image with a bounding box; And / or, if the detection result indicates that the original image includes the target visual element and is an image in the video to be reviewed, the timecode of the original image is recorded.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images