Image detection method, storage medium and computer terminal

Abstract

The application discloses an image detection method, a storage medium and a computer terminal. The method comprises the following steps: obtaining a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the resolutions of the multiple feature maps are different; performing feature fusion on each feature map and a historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map is extracted; and performing image detection on the target feature map to obtain a target detection result of a target object contained in the target image. The application solves the technical problem of low efficiency of image detection in the related art.

Description

Technical Field

[0001] This application relates to the field of image detection, and more specifically, to an image detection method, a storage medium, and a computer terminal. Background Technology

[0002] Image detection is a computer vision task used to predict the class labels and bounding box locations of all objects of interest in an image. Traditional object detectors use many handcrafted components, such as providing prior object sizes, heuristic object matching rules, and non-maximum suppression algorithms to remove duplicate results. These components complicate the overall image detection process. To reduce this complexity, end-to-end object detectors are generally adopted, characterized by an encoder-decoder architecture. The encoder is used for feature fusion, and the decoder is used for object lookup. However, the encoder incurs significant computational overhead during feature fusion to extract high-resolution low-scale feature maps, resulting in low efficiency in image detection.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an image detection method, a storage medium, and a computer terminal to at least solve the technical problem of low efficiency in image detection in related technologies.

[0005] According to one aspect of the embodiments of this application, an image detection method is provided, comprising: acquiring a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with a historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; and performing image detection on the target feature map to obtain a target detection result of a target object contained in the target image.

[0006] According to another aspect of the embodiments of this application, an image detection method is also provided, comprising: a cloud server receiving a target image uploaded by a client; the cloud server performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; the cloud server performing feature fusion between each feature map and a historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; the cloud server performing image detection on the target feature map to obtain a target detection result of a target object contained in the target image; and the cloud server sending the target detection result to the client.

[0007] According to another aspect of the embodiments of this application, an image detection method is also provided, comprising: receiving a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with a historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain a target detection result of a target object contained in the target image; and outputting the target detection result.

[0008] According to another aspect of the embodiments of this application, an image detection method is also provided, comprising: displaying a target image on an interactive interface; if a target detection command is detected in any region of the interactive interface, triggering multiple feature extractions on the target image to obtain multiple feature maps, and fusing each feature map with a historical feature map corresponding to each feature map to obtain a target feature map, wherein the multiple feature maps have different resolutions, and the historical feature maps are feature maps that have been fused before each feature map was extracted; and displaying the target detection result of the target object contained in the target image on the interactive interface, wherein the target detection result is obtained by performing image detection on the target feature map.

[0009] According to another aspect of the embodiments of this application, an image detection method is also provided, comprising: acquiring a target image upon receiving a communication request; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with a historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain a target detection result of a target object contained in the target image; and displaying the target detection result in a communication interface.

[0010] According to another aspect of the embodiments of this application, an image detection method is also provided, comprising: acquiring a target image by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with a historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain a target detection result of a target object contained in the target image; and outputting the target detection result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the target detection result.

[0011] According to one aspect of the embodiments of this application, an image detection apparatus is provided, comprising: an acquisition module for acquiring a target image; an extraction module for performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; a fusion module for fusing each feature map with a corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; and a detection module for performing image detection on the target feature map to obtain a detection result of the target image.

[0012] According to one aspect of the embodiments of this application, another image detection apparatus is provided, comprising: a receiving module for receiving a target image uploaded by a client via a cloud server; an extraction module for performing multiple feature extractions on the target image via the cloud server to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; a fusion module for fusing each feature map with a corresponding historical feature map via the cloud server to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; a detection module for performing image detection on the target feature map via the cloud server to obtain a target detection result of a target object contained in the target image; and a sending module for sending the target detection result to the client via the cloud server.

[0013] According to one aspect of the embodiments of this application, another image detection apparatus is provided, comprising: a receiving module for receiving a target image; an extraction module for performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; a fusion module for fusing each feature map with a historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; a detection module for performing image detection on the target feature map to obtain a target detection result of a target object contained in the target image; and an output module for outputting the target detection result.

[0014] According to one aspect of the embodiments of this application, another image detection apparatus is provided, comprising: a display module for displaying a target image on an interactive interface; a trigger module for triggering multiple feature extractions on the target image to obtain multiple feature maps if a target detection command is detected in any region of the interactive interface, and fusing each feature map with a corresponding historical feature map to obtain a target feature map, wherein the multiple feature maps have different resolutions, and the historical feature maps are feature maps that have been fused before each feature map was extracted; and a display module for displaying the target detection result of a target object contained in the target image on the interactive interface, wherein the target detection result is obtained by performing image detection on the target feature map.

[0015] According to one aspect of the embodiments of this application, another image detection apparatus is provided, comprising: an acquisition module, configured to acquire a target image upon receiving a communication request; an extraction module, configured to perform multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; a fusion module, configured to fuse each feature map with a corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; a detection module, configured to perform image detection on the target feature map to obtain a target detection result of a target object contained in the target image; and a display module, configured to display the target detection result in a communication interface.

[0016] According to one aspect of the embodiments of this application, another image detection apparatus is provided, comprising: a first calling module for acquiring a target image by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the target image; an extraction module for performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; a fusion module for fusing each feature map with a historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; a detection module for performing image detection on the target feature map to obtain a target detection result of a target object contained in the target image; and a second calling module for outputting the target detection result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the target detection result.

[0017] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to perform the above-described image detection method.

[0018] According to another aspect of the embodiments of this application, a computer terminal is also provided, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described image detection method when it runs.

[0019] In this embodiment, a target image is first acquired. Multiple feature extractions are performed on the target image to obtain multiple feature maps with different resolutions. Then, each feature map is fused with its corresponding historical feature map to obtain a target feature map. The historical feature maps are those fused before each feature map was extracted. Finally, image detection is performed on the target feature map to obtain the target object detection result, thus improving image detection efficiency. It is noteworthy that after feature extraction, the feature map can be fused with the previous historical feature map. The feature extraction and feature fusion stages proceed in parallel, with the feature fusion step performed simultaneously during feature extraction. By the time the final feature fusion stage is reached, the obtained target feature map has fully incorporated the previous features. Therefore, image detection using this target feature map can improve the accuracy of the target detection result. Furthermore, by fusing each feature map with its corresponding historical feature map, cross-scale feature fusion is achieved. This process does not require the use of the encoder's feature fusion function, reducing computational complexity and thus solving the technical problem of low image detection efficiency in related technologies. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image detection method according to existing technology;

[0022] Figure 2 This is a flowchart of an image detection method according to Embodiment 1 of this application;

[0023] Figure 3 This is a flowchart of another image detection method according to Embodiment 1 of this application;

[0024] Figure 4 This is a schematic diagram of another image detection structure according to Embodiment 1 of this application;

[0025] Figure 5 This is a flowchart of an image detection method according to Embodiment 2 of this application;

[0026] Figure 6 This is a flowchart of an image detection method according to Embodiment 3 of this application;

[0027] Figure 7This is a flowchart of an image detection method according to Embodiment 4 of this application;

[0028] Figure 8 This is a schematic diagram of an interactive interface according to Embodiment 4 of this application;

[0029] Figure 9 This is a flowchart of an image detection method according to Embodiment 5 of this application;

[0030] Figure 10 This is a flowchart of an image detection method according to Embodiment 6 of this application;

[0031] Figure 11 This is a schematic diagram of an image detection device according to Embodiment 7 of this application;

[0032] Figure 12 This is a schematic diagram of an image detection device according to Embodiment 8 of this application;

[0033] Figure 13 This is a schematic diagram of an image detection device according to Embodiment 9 of this application;

[0034] Figure 14 This is a schematic diagram of an image detection device according to Embodiment 10 of this application;

[0035] Figure 15 This is a schematic diagram of an image detection device according to Embodiment 11 of this application;

[0036] Figure 16 This is a schematic diagram of an image detection device according to Embodiment 12 of this application;

[0037] Figure 17 This is a structural block diagram of a computer terminal according to an embodiment of this application. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0041] Transformer: Self-Attention Transformer Network, a deep learning model based on the attention mechanism.

[0042] Transformer Encoder: A self-attention transform network encoder, or simply encoder, used in object detection tasks for further information interaction and mixing of image features.

[0043] Transformer Decoder: A self-attention transform network decoder, or simply decoder, is used in object detection tasks to extract object information from image features and convert it into object detection results.

[0044] DETR: An encoder-decoder based object detection paradigm, characterized by end-to-end detection, meaning the detection result is the final result, without the need for deduplication or other post-processing operations.

[0045] The aforementioned DETR is an end-to-end object detector characterized by an encoder-decoder architecture. The encoder is used for feature fusion, and the decoder is used for object lookup. DETR achieves good detection results without using handcrafted components. However, DETR suffers from several problems, such as slow training convergence, poor performance on small objects, and high computational complexity. A series of subsequent works have aimed to address these issues, such as accelerating training convergence through novel cross-attention techniques and improving accuracy on small objects through multi-scale feature maps. While these works have made some progress, the problem of high computational complexity remains unresolved.

[0046] Computational complexity is a theoretical metric for evaluating the speed of deep learning models. It represents the upper limit of the running speed of an optimized model deployed on hardware and has significant reference value. Researching high-precision, low-computational-complexity end-to-end object detection helps improve the operational efficiency in object detection scenarios. Currently, the detection speed of models is improved through the following methods:

[0047] Deformable DETR (Deformable Attention Target Detector): Introduces a deformable attention mechanism that focuses only on a small number of sampled points predicted from the features of the query, which greatly speeds up the model training convergence and reduces memory overhead, thus enabling the use of multi-scale feature maps to improve detection accuracy.

[0048] Efficient DETR (Target Detector with Dense Target Prediction Head): Introduces a dense target prediction head for better initialization of the decoder's target query elements.

[0049] TSP (Transformer-based Set Prediction): Uses a traditional object detection head instead of a decoder, and directly uses the encoder's output for object prediction.

[0050] Conditional DETR (Target Detector with Cross-Attention Mechanism): This proposes a conditional cross-attention mechanism that predicts a spatial embedding from each output of the previous decoder layer and feeds it back to the next cross-attention calculation, enabling the content query to be located in different regions.

[0051] SMCA (Spatial Modulation Cooperative Attention): Uses location-restricted regression loss to force attention to the vicinity of the initially estimated bounding box location.

[0052] The main purpose of the above method is to solve the problem of slow convergence speed caused by cross-attention. However, its drawback is that it does not change the feature fusion process of the encoder-decoder framework, and the encoder needs to pay a lot of computational overhead to extract high-resolution low-scale feature maps.

[0053] To address the aforementioned issues, this application provides an image detection method. By using a Transformer backbone network to provide intra-scale feature interaction and designing a cross-scale attention mechanism within the backbone network to provide cross-scale feature interaction, the feature fusion function is integrated into the backbone network. Furthermore, the encoder is removed, thereby reducing computational complexity.

[0054] Example 1

[0055] According to an embodiment of this application, an embodiment of an image detection method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0056] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an image detection method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0057] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). This data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0058] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the image detection method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0059] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0060] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0061] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance and is intended to illustrate the types of components that may exist in the aforementioned computer device (or mobile device).

[0062] Under the aforementioned operating environment, this application provides the following: Figure 2 The image detection method shown. Figure 2 This is a flowchart of an image detection method according to an embodiment of this application.

[0063] Step S202: Obtain the target image.

[0064] The target image mentioned above can be an image to be predicted or an image used for training. The target image may contain target objects, which can be objects to be detected, and there can be one or more target objects. The target image can be a remote sensing image, radar image, or satellite image, but is not limited to these.

[0065] In one optional embodiment, the target image can be a building image, the target object can be a building, and the building category can be obtained by detecting the target image; the target image can be a traffic image, the target object can be a vehicle, and the vehicle category can be obtained by detecting the traffic image; the target image can be a face image, the target object can be a face, and the face information can be obtained by detecting the face image.

[0066] In another optional embodiment, the target image can be displayed within the image acquisition area, and the feedback area can display the target detection results of the target object contained in the target image. Users can actively upload the target image to the server for processing. Users can upload the target image to the server by clicking the "Upload Image" button in the interactive interface or by dragging the target image directly into the dotted box. The uploaded target image can be displayed within the image acquisition area. The server here can be a locally deployed server or a cloud-based server.

[0067] Step S204: Perform feature extraction on the target image multiple times to obtain multiple feature maps.

[0068] Among them, several feature maps have different resolutions.

[0069] In one optional embodiment, in order to improve the accuracy of the target object detection result in the target image, the target image can be subjected to multiple feature extractions to obtain multiple feature maps. Since the resolution of the multiple feature maps is different, the accuracy of the target detection result can be improved by using multiple feature maps to detect the target object in the target image.

[0070] In another alternative embodiment, Hi can be the i-th extraction stage of the visual Transformer, xi can be the i-th feature map, j = i-1, and the calculation process of feature map xi is xi = Hi(xj). Each patch (size) of feature map xi has a global receptive field within the scale on xi.

[0071] It should be noted that the specific implementation of Hi is not restricted; in practice, attention algorithms with computational complexity proportional to the input resolution (such as PVT) are often chosen.

[0072] Step S206: Perform feature fusion between each feature map and the corresponding historical feature map to obtain the target feature map.

[0073] Among them, the historical feature map is the feature map that has been fused before each feature map is extracted.

[0074] The aforementioned number of historical feature maps can be one or more feature maps that have already been fused. These historical feature maps may also include the feature maps extracted the first time.

[0075] In an optional embodiment, when a feature map is extracted, it can be sent to the fusion stage, where the feature map is fused with historical feature maps. Multiple fusion stages and multiple feature extraction stages are carried out in parallel and upward. In the fusion stage, information from low-level feature maps can be aggregated into high-level feature maps through spatial reduction, ultimately obtaining the target feature map mentioned above.

[0076] In another alternative embodiment, SA(q,k) can be an attention operation, where q is a query element and k is a key element. Let xi' be the i-th fused feature map. The calculation process of the fused feature map xi' is xi'=SA(xi,[x1',x2',...,xj',xi]), and each patch of the fused feature map xi' has a cross-scale receptive field on x1',x2',...,xj'.

[0077] It should be noted that the specific implementation of SA is not restricted, and in practice, attention algorithms with computational complexity proportional to the input resolution (such as PVT) are often chosen.

[0078] Step S208: Perform image detection on the target feature map to obtain the target detection result of the target object contained in the target image.

[0079] In an optional embodiment, after the last fusion stage is completed, the target feature map can be directly sent to the decoder for decoding to obtain the target detection result of the target object contained in the target image. The decoder can be a deformable decoder, and the backbone network corresponding to the feature extraction process and the image detection process can be assembled with any decoder to form an end-to-end target detector.

[0080] In another alternative embodiment, feature extraction and feature fusion functions can be provided through a backbone network. The backbone network can be a Transformer, which can provide cross-scale attention mechanisms to enable cross-scale feature interaction. This allows the feature fusion function to be integrated into the backbone network, and the encoder can be further removed to reduce computational complexity.

[0081] Through the above steps, the target image is first acquired, and multiple feature extractions are performed on the target image to obtain multiple feature maps with different resolutions. Then, each feature map is fused with its corresponding historical feature map to obtain the target feature map. The historical feature maps are those fused before each feature map was extracted. Finally, image detection is performed on the target feature map to obtain the target object detection result, thus improving image detection efficiency. It is noteworthy that after feature extraction, the feature map can be fused with the previous historical feature map. The feature extraction and feature fusion stages proceed in parallel, with the feature fusion step performed simultaneously during feature extraction. By the time the final feature fusion stage is reached, the obtained target feature map has fully incorporated the previous features. Therefore, image detection using this target feature map can improve the accuracy of the target detection result. Furthermore, by fusing each feature map with its corresponding historical feature map, cross-scale feature fusion is achieved. This process does not require the use of the encoder's feature fusion function, reducing computational complexity and thus solving the technical problem of low efficiency in image detection in related technologies.

[0082] In the above embodiments of this application, performing multiple feature extractions on the target image to obtain multiple feature maps includes: inputting the target image into a feature extraction model and obtaining multiple feature maps output by the feature extraction model. The feature extraction model includes multiple feature extraction layers connected in sequence, each feature extraction layer being used to extract features from the input first image and output a feature map corresponding to the first image.

[0083] The feature extraction model described above can be a Transformer. The first image above is the feature map output by the previous feature extraction layer.

[0084] In one optional embodiment, the target image can be input into the first feature extraction layer of the feature extraction model, and then the feature map corresponding to the target image can be output. Then, the feature map corresponding to the target image can be used as the first image and input into the next feature extraction layer connected to the first feature extraction layer, outputting the feature map corresponding to the first image. This feature map can then be used as the first image and input into the next feature extraction layer connected in sequence, until the last feature extraction layer outputs the feature map corresponding to the first image.

[0085] It should be noted that when each feature extraction layer outputs the feature map corresponding to the first image, this feature map can be fused with all historical feature maps. When the last feature extraction layer outputs the feature map, this feature map can be fused with the previously fused feature maps, which can improve the accuracy of the obtained target feature map.

[0086] In the above embodiments of this application, feature fusion of each feature map with its corresponding historical feature map to obtain a target feature map includes: inputting multiple feature maps into a feature fusion model and obtaining the target feature map output by the feature fusion model. The feature fusion model includes: multiple feature fusion layers connected in sequence, with each feature fusion layer and multiple feature extraction layers connected in a one-to-one correspondence. Each feature fusion layer is connected to all other feature fusion layers. Each feature fusion layer is used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers and output the fused feature map.

[0087] In one optional embodiment, when each feature extraction layer outputs a feature map, the feature map can be input into the feature fusion model. Specifically, the feature map output by the first feature extraction layer can be input into the first feature fusion layer of the feature fusion model, and the feature map output by the second feature extraction layer, which is sequentially connected to the first feature extraction layer, can be input into the second feature fusion layer of the feature fusion model. The second feature fusion layer can fuse the feature map with the feature map in the first feature fusion layer to obtain a fused feature map. When each feature fusion layer receives the feature map output by the feature extraction model, it can fuse the feature map with the features fused in the previous feature fusion layer to improve the accuracy of the feature map.

[0088] It should be noted that the target feature map is the feature map obtained by fusing the fused feature maps from each feature fusion layer with the currently acquired feature map.

[0089] In the above embodiments of this application, each feature fusion layer is used to perform attention operations on the first image and the feature map that has been fused to generate a fused feature map.

[0090] The aforementioned attention operation refers to the aggregation of information from low-level feature maps into high-level feature maps through a spatial reduction attention mechanism. This results in lower resolution in the high-level feature maps, meaning fewer query elements, which is more conducive to reducing computational overhead and thus improving fusion efficiency.

[0091] In the above embodiments of this application, image detection of the target feature map to obtain the target detection result of the target object contained in the target image includes: inputting the target feature map into the decoder and obtaining the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object and the second score of the target category; filtering the predicted position based on the first score and the second score to obtain the filtered predicted position; and generating the target detection result based on the filtered predicted position and the target category.

[0092] The decoder types mentioned above are not restricted; for example, they can be deformable decoders.

[0093] The predicted locations mentioned above can be multiple.

[0094] The predicted location of the target object mentioned above refers to the region where the predicted target object is located in the target image. The first score of the predicted location can be obtained based on the marked location in the target image and the predicted location. The first score can represent the prediction accuracy of the predicted location.

[0095] The target category of the target object mentioned above can be the classification to which the target object belongs. The second score of the target category can be obtained based on the classification labeled in the target image and the target classification. The second score can represent the prediction accuracy of the target classification.

[0096] In an optional embodiment, since the target feature map is a fused feature map, after obtaining the target feature map, it can be directly input into the decoder, and the initial detection result output by the decoder can be obtained. The predicted position can be scored according to the product of the first score and the second score, and the predicted position can be filtered based on the score to obtain the predicted position with higher accuracy after filtering. Based on the predicted position with higher accuracy after filtering and the target category, a target detection result with higher accuracy can be generated, thereby improving the accuracy of the target detection result.

[0097] In another optional embodiment, during the training process, the training feature map can be input into the decoder, and the training detection result output by the decoder can be obtained. The training detection result may include: the predicted position of the target object, the first score of the predicted position, the target category of the target object, and the second score of the target category. The predicted position can be filtered according to the first score and the second score in the training detection result to obtain the filtered predicted position. Based on the filtered predicted position and the target category, the target detection result is generated.

[0098] In the above embodiments of this application, filtering the predicted position based on the first score and the second score to obtain the filtered predicted position includes: obtaining the product of the first score and the second score to obtain the predicted score of the predicted position; and filtering the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0099] The predicted location mentioned above can be the position of the predicted bounding box on the target object.

[0100] In one optional embodiment, the product of the first score and the second score can be obtained to get the predicted score of the predicted position. Predicted positions with lower predicted scores can be filtered by a preset threshold to obtain predicted positions with higher predicted scores, thereby improving the accuracy of the predicted position.

[0101] In the above embodiments of this application, the method further includes: obtaining target annotation results of the target object and mask annotation results of the target image, wherein the mask annotation results are used to characterize whether each image pixel in the target image belongs to the target object; constructing a first loss function based on the target annotation results and the target detection results; performing image recognition on the target image based on each feature map to obtain the mask detection results corresponding to each feature pixel in each feature map; constructing a second loss function based on the mask annotation results and the mask detection results; and adjusting the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0102] The above target annotation results can be the result of pre-annotating the target object.

[0103] The above mask annotation results can be used to indicate whether a pixel belongs to the target object by annotating the pixel.

[0104] In an optional embodiment, target annotation results and target detection results of the target object can be obtained. Specifically, the error between the annotation position and the predicted position can be determined based on the cross-union ratio between the annotation position in the target annotation result and the predicted position in the target detection result. The first loss function can be constructed by combining the error with the category of the target object detected in the target detection result.

[0105] In another alternative embodiment, a second loss function can be constructed based on the difference between the mask annotation result and the mask detection result. Specifically, the sub-mask annotation result corresponding to the feature map can be obtained by scaling the mask annotation result. Each sub-mask annotation result can be calculated with the mask detection result corresponding to each feature map to obtain the loss function corresponding to each feature map. Multiple loss functions can be combined to obtain the second loss function.

[0106] In another alternative embodiment, the model parameters of the feature extraction model, the feature fusion model, and the decoder are adjusted according to the first loss function and the second loss function, thereby improving the accuracy of the feature extraction model, the feature fusion model, and the decoder.

[0107] In the above embodiments of this application, constructing a first loss function based on target annotation results and target detection results includes: determining the intersection-union ratio (IU) of the annotation positions in the target annotation results and the detection positions in the target detection results to obtain the position IU; and constructing a first loss function based on the position IU and the score of the detection category in the target detection results.

[0108] In an alternative embodiment, the first loss function can be constructed using the following formula:

[0109]

[0110] Among them, L awr Let B be the first loss function, and B be the total number of predicted boxes. i It is the i-th predicted position. It is the labeled position corresponding to the i-th predicted position. It is the detection category of the target object. To calculate the intersection-union ratio between the predicted and labeled locations, BCE is the binary cross-entropy loss function.

[0111] In another alternative embodiment, the first loss function can suppress detection results with low intersection between predicted and labeled locations during the detection process, thereby improving the accuracy of the predicted location.

[0112] In the above embodiments of this application, constructing a second loss function based on the mask annotation result and the mask detection result includes: scaling the mask annotation result based on the size of each feature map to obtain a sub-mask annotation result corresponding to each feature map; obtaining a sub-loss function corresponding to each feature map based on the sub-mask annotation result and the mask detection result corresponding to each feature map; and obtaining the sum of the sub-loss functions corresponding to multiple feature maps to obtain the second loss function.

[0113] In an alternative embodiment, the second loss function can be constructed using the following formula:

[0114]

[0115] Among them, L token Let t be the second loss function. j [p, q] represents the mask annotation result of the pixel at position (p, q) in the feature map, FFN(x i[p, q]) represents the mask detection results corresponding to the feature maps, N is the number of feature maps, and Focal is the improved binary classification cross-entropy function.

[0116] In another optional embodiment, the mask annotation result can be obtained by scaling each feature map. Specifically, the mask annotation result can be scaled according to the size of each feature map to obtain the sub-mask annotation result corresponding to each feature map. Specifically, the mask annotation result can be scaled using the following formula to obtain the mask annotation result corresponding to each feature map:

[0117] t j =Interpolate(M, SazeOf(x) j ))

[0118] Where M is the mask annotation result, SazeOf(x j ) represents the scaling size of the feature map, t j The mask annotation results are for each feature map, and Interpolate is the sampling function.

[0119] In another alternative embodiment, each size of feature map can be used to classify the content of the corresponding region on the input image, and supervised training can be performed using mask annotation to improve the extraction accuracy of the feature maps.

[0120] The following is combined with Figure 3 and Figure 4 A preferred embodiment of this application will be described in detail. This method can be executed by a mobile terminal or a server. In this embodiment, the method is described as being executed by a server. Figure 3 As shown, the method may include the following steps:

[0121] Step S301: The target image is input into the backbone network, and after multiple feature extraction stages, multiple feature maps with progressively decreasing resolution are obtained.

[0122] Step S302: The generated feature map can be fused with the historical feature map through multiple feature fusion stages that run in parallel with the feature extraction stage.

[0123] Optional, such as Figure 4 As shown, parallel to the extraction stage are multiple feature fusion stages, each feature... Figure 1 Once generated, the feature map is immediately sent to the fusion stage. The feature map itself acts as a query element, and all previously fully fused feature maps can be densely connected to the current fusion stage as additional key elements. In the fusion stage, information from low-level feature maps is gathered into high-level feature maps through a spatial reduction attention mechanism.

[0124] In step S303, after the feature fusion is completed in the last fusion stage, the obtained target feature map can be sent to the decoder for decoding to obtain the prediction result of the target object in the target image.

[0125] Optionally, after the final fusion stage, the feature map is directly sent to the decoder for decoding to obtain the final prediction result. For the image detection process, no encoder is involved throughout, and the decoder type is unrestricted (e.g., deformable decoder). The backbone network can assemble any decoder to form an end-to-end object detector.

[0126] By combining the above steps Figure 4 Feature fusion can be further divided into intra-scale feature interaction and cross-scale feature interaction. Intra-scale feature interaction is provided by using a Transformer backbone network, and cross-scale feature interaction is provided by designing a cross-scale attention mechanism within the backbone network. This integrates the feature fusion function into the backbone network and further removes the encoder, thereby reducing computational complexity. Specifically, the image can be input into extraction stage 1 to obtain features. Figure 1 , will feature Figure 1 In the input fusion stage, features are processed in extraction stage 2. Figure 1 Extraction is performed to obtain features. Figure 2 , will feature Figure 2 The features are input into fusion stage 1, where they are combined. Figure 1 and characteristics Figure 2 In stage 3 of the extraction process, features are fused together. Figure 2 Extraction is performed to obtain features. Figure 3 In fusion phase 2, the feature maps and features fused in fusion phase 1 are combined. Figure 1 The features are then fused. In extraction stage 4, the features are... Figure 3 Extraction is performed to obtain features. Figure 4 , will feature Figure 4 The input is fed into fusion stage 3, where the features are processed. Figure 1 The feature maps fused in fusion stage 1 and fusion stage 2 are fused together to generate a target feature map. The target feature map is then input into the decoder for detection to obtain the decoding result.

[0127] In another alternative embodiment, extracting low-scale feature maps from high-resolution images via the encoder incurs significant computational overhead. For example, inputting a 1280x800 image typically requires feature maps at four scales: 160x100, 80x50, 40x25, and 20x12. During feature fusion in the encoder, the top-level feature map has 240 query elements, while the lowest-level feature map has 16,000 query elements, resulting in substantial computational overhead during attention operations. Therefore, the encoder can be omitted, and instead, a computationally efficient cross-scale attention mechanism is added to the backbone network, forming a pure decoder end-to-end object detector. This attention mechanism prioritizes utilizing high-level feature maps, with information from low-level feature maps being aggregated into high-level feature maps through spatial reduction. The lower resolution of high-level feature maps (e.g., 20x12 feature maps), meaning fewer query elements, further reduces computational overhead.

[0128] Furthermore, by following the steps described above, the use of an encoder-decoder framework can be avoided, and a pure decoder end-to-end object detector is proposed, which can significantly reduce computational complexity. In addition, a computationally efficient cross-scale attention mechanism is proposed and attached to the Transformer backbone network, which enables the backbone network to directly generate fully fused feature maps. This approach features low computational overhead and high detection accuracy.

[0129] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0131] Example 2

[0132] According to an embodiment of this application, an embodiment of an image detection method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0133] Figure 5 This is a flowchart of an image detection method according to an embodiment of the present invention. Figure 5 As shown, the method may include the following steps:

[0134] In step S502, the cloud server receives the target image uploaded by the client.

[0135] The aforementioned client and cloud server can interact via a specific interface. The client can pass the target image selected by the user to the interface function as a parameter of the interface, thereby achieving the purpose of uploading the target image to the cloud server.

[0136] In step S504, the cloud server performs multiple feature extractions on the target image to obtain multiple feature maps.

[0137] Among them, several feature maps have different resolutions.

[0138] In step S506, the cloud server performs feature fusion between each feature map and the corresponding historical feature map to obtain the target feature map.

[0139] Among them, the historical feature map is the feature map that has been fused before each feature map is extracted.

[0140] Step S508: The cloud server performs image detection on the target feature map to obtain the target detection result of the target object contained in the target image.

[0141] Step S510: The cloud server sends the target detection results to the client.

[0142] In the above embodiments of this application, the cloud server performs multiple feature extractions on the target image to obtain multiple feature maps, including: the cloud server inputs the target image into the feature extraction model and obtains multiple feature maps output by the feature extraction model, wherein the feature extraction model includes: multiple feature extraction layers connected in sequence, each feature extraction layer is used to extract features from the input first image and output the feature map corresponding to the first image.

[0143] In the above embodiments of this application, the cloud server performs feature fusion on each feature map and the corresponding historical feature map to obtain the target feature map. This includes: the cloud server inputting multiple feature maps into the feature fusion model and obtaining the target feature map output by the feature fusion model. The feature fusion model includes: multiple feature fusion layers connected in sequence, with each feature fusion layer and multiple feature extraction layers connected in a one-to-one correspondence. Each feature fusion layer is connected to all other feature fusion layers. Each feature fusion layer is used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers, and outputs the fused feature map.

[0144] In the above embodiments of this application, each feature fusion layer is used to perform attention operations on the first image and the feature map that has been fused to generate a fused feature map.

[0145] In the above embodiments of this application, the cloud server performs image detection on the target feature map to obtain the target detection result of the target object contained in the target image, including: the cloud server inputs the target feature map to the decoder and obtains the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object, and the second score of the target category; the cloud server filters the predicted position based on the first score and the second score to obtain the filtered predicted position; the cloud server generates the target detection result based on the filtered predicted position and the target category.

[0146] In the above embodiments of this application, the cloud server filters the predicted position based on a first score and a second score to obtain the filtered predicted position, including: the cloud server obtaining the product of the first score and the second score to obtain the predicted score of the predicted position; the cloud server filtering the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0147] In the above embodiments of this application, the method further includes: a cloud server acquiring target annotation results of the target object and mask annotation results of the target image, wherein the mask annotation results are used to characterize whether each image pixel in the target image belongs to the target object; the cloud server constructing a first loss function based on the target annotation results and the target detection results; performing image recognition on the target image based on each feature map to obtain the mask detection results corresponding to each feature pixel in each feature map; the cloud server constructing a second loss function based on the mask annotation results and the mask detection results; and the cloud server adjusting the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0148] In the above embodiments of this application, the cloud server constructs a first loss function based on the target annotation results and the target detection results, including: the cloud server determines the intersection-union ratio (IU) of the labeled positions in the target annotation results and the detection positions in the target detection results to obtain the position IU; the cloud server constructs a first loss function based on the position IU and the score of the detection category in the target detection results.

[0149] In the above embodiments of this application, the cloud server constructs a second loss function based on the mask annotation results and mask detection results, including: scaling the mask annotation results based on the size of each feature map to obtain a sub-mask annotation result corresponding to each feature map; obtaining a sub-loss function corresponding to each feature map based on the sub-mask annotation results and the mask detection results corresponding to each feature map; and obtaining the sum of the sub-loss functions corresponding to multiple feature maps to obtain the second loss function.

[0150] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0151] Example 3

[0152] According to an embodiment of this application, an embodiment of an image detection method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order.

[0153] Figure 6 This is a flowchart of an image detection method according to an embodiment of the present invention, such as... Figure 6 As shown, the method may include the following steps:

[0154] Step S602: Receive the target image.

[0155] Step S604: Perform multiple feature extractions on the target image to obtain multiple feature maps.

[0156] Among them, several feature maps have different resolutions.

[0157] Step S606: Perform feature fusion between each feature map and the historical feature map corresponding to each feature map to obtain the target feature map.

[0158] Among them, the historical feature map is the feature map that has been fused before each feature map is extracted.

[0159] Step S608: Perform image detection on the target feature map to obtain the target detection result of the target object contained in the target image.

[0160] Step S610: Output the target detection results.

[0161] In the above embodiments of this application, performing multiple feature extractions on the target image to obtain multiple feature maps includes: inputting the target image into a feature extraction model and obtaining multiple feature maps output by the feature extraction model. The feature extraction model includes multiple feature extraction layers connected in sequence, each feature extraction layer being used to extract features from the input first image and output a feature map corresponding to the first image.

[0162] In the above embodiments of this application, feature fusion of each feature map with its corresponding historical feature map to obtain a target feature map includes: inputting multiple feature maps into a feature fusion model and obtaining the target feature map output by the feature fusion model. The feature fusion model includes: multiple feature fusion layers connected in sequence, with each feature fusion layer and multiple feature extraction layers connected in a one-to-one correspondence. Each feature fusion layer is connected to all other feature fusion layers. Each feature fusion layer is used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers and output the fused feature map.

[0163] In the above embodiments of this application, each feature fusion layer is used to perform attention operations on the first image and the feature map that has been fused to generate a fused feature map.

[0164] In the above embodiments of this application, image detection of the target feature map to obtain the target detection result of the target object contained in the target image includes: inputting the target feature map into the decoder and obtaining the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object and the second score of the target category; filtering the predicted position based on the first score and the second score to obtain the filtered predicted position; and generating the target detection result based on the filtered predicted position and the target category.

[0165] In the above embodiments of this application, filtering the predicted position based on the first score and the second score to obtain the filtered predicted position includes: obtaining the product of the first score and the second score to obtain the predicted score of the predicted position; and filtering the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0166] In the above embodiments of this application, the method further includes: obtaining target annotation results of the target object and mask annotation results of the target image, wherein the mask annotation results are used to characterize whether each image pixel in the target image belongs to the target object; constructing a first loss function based on the target annotation results and the target detection results; performing image recognition on the target image based on each feature map to obtain the mask detection results corresponding to each feature pixel in each feature map; constructing a second loss function based on the mask annotation results and the mask detection results; and adjusting the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0167] In the above embodiments of this application, constructing a first loss function based on target annotation results and target detection results includes: determining the intersection-union ratio (IU) of the annotation positions in the target annotation results and the detection positions in the target detection results to obtain the position IU; and constructing a first loss function based on the position IU and the score of the detection category in the target detection results.

[0168] In the above embodiments of this application, constructing a second loss function based on the mask annotation result and the mask detection result includes: scaling the mask annotation result based on the size of each feature map to obtain a sub-mask annotation result corresponding to each feature map; obtaining a sub-loss function corresponding to each feature map based on the sub-mask annotation result and the mask detection result corresponding to each feature map; and obtaining the sum of the sub-loss functions corresponding to multiple feature maps to obtain the second loss function.

[0169] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0170] Example 4

[0171] According to an embodiment of this application, an embodiment of an image detection method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order.

[0172] Figure 7 This is a flowchart of an image detection method according to an embodiment of the present invention, such as... Figure 7 As shown, the method may include the following steps:

[0173] Step S702: Display the target image on the interactive interface.

[0174] The aforementioned interactive interface can be an operation interface provided to the user on the display screen of the device used to perform target image processing. For example, the interactive interface can be an operation interface displayed on the display screen of a mobile terminal or an operation interface displayed on the display screen of a computer terminal, but it is not limited to these.

[0175] In one alternative embodiment, after acquiring the target image, the user can click as follows: Figure 8 The interactive interface shown includes an "Upload Image" button, or you can drag the target image to the dotted box to select the target image, which can then be displayed as shown. Figure 8 The first display area shown.

[0176] In another alternative embodiment, the user can click as follows: Figure 8 The "Image Detection" button shown allows for multiple feature extractions of the target object contained in the target image, resulting in multiple feature maps. Each feature map is then fused with its corresponding historical feature map to obtain a target feature map. This target feature map is then used for detection to obtain the target detection result of the target object contained in the target image.

[0177] Step S704: If a target detection command is detected in any area of ​​the interactive interface, multiple feature extractions are triggered on the target image to obtain multiple feature maps. Each feature map is then fused with the corresponding historical feature map to obtain the target feature map.

[0178] Among them, multiple feature maps have different resolutions, and historical feature maps are feature maps that have been fused before each feature map is extracted.

[0179] The target detection instruction in the above steps can be an instruction generated by the user clicking a specific button on the interactive interface, or an instruction generated by the user performing a predetermined gesture operation on the interactive interface. This instruction is used to extract features from the target image multiple times to obtain multiple feature maps, and to fuse each feature map with the historical feature map corresponding to each feature map to obtain a target feature map. Based on the target feature map, the target detection result of the target object contained in the target image is obtained.

[0180] In one alternative embodiment, after uploading the target image, the user can click as follows: Figure 8 The "Image Detection" button shown generates a target detection command, or a target detection command can be generated directly through gesture operation. The computer terminal, mobile terminal, or server can receive the target detection command, obtain the target feature map corresponding to the target image based on the target detection command, and obtain the target detection result of the target object contained in the target image based on the target feature map.

[0181] Step S706: Display the target detection results of the target objects contained in the target image on the interactive interface.

[0182] The target detection result is obtained by performing image detection on the target feature map.

[0183] In an optional embodiment, the target detection results can be displayed as follows: Figure 8 The second display area shown.

[0184] In the above embodiments of this application, performing multiple feature extractions on the target image to obtain multiple feature maps includes: inputting the target image into a feature extraction model and obtaining multiple feature maps output by the feature extraction model. The feature extraction model includes multiple feature extraction layers connected in sequence, each feature extraction layer being used to extract features from the input first image and output a feature map corresponding to the first image.

[0185] In the above embodiments of this application, feature fusion of each feature map with its corresponding historical feature map to obtain a target feature map includes: inputting multiple feature maps into a feature fusion model and obtaining the target feature map output by the feature fusion model. The feature fusion model includes: multiple feature fusion layers connected in sequence, with each feature fusion layer and multiple feature extraction layers connected in a one-to-one correspondence. Each feature fusion layer is connected to all other feature fusion layers. Each feature fusion layer is used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers and output the fused feature map.

[0186] In the above embodiments of this application, each feature fusion layer is used to perform attention operations on the first image and the feature map that has been fused to generate a fused feature map.

[0187] In the above embodiments of this application, image detection of the target feature map to obtain the target detection result of the target object contained in the target image includes: inputting the target feature map into the decoder and obtaining the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object and the second score of the target category; filtering the predicted position based on the first score and the second score to obtain the filtered predicted position; and generating the target detection result based on the filtered predicted position and the target category.

[0188] In the above embodiments of this application, filtering the predicted position based on the first score and the second score to obtain the filtered predicted position includes: obtaining the product of the first score and the second score to obtain the predicted score of the predicted position; and filtering the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0189] In the above embodiments of this application, the method further includes: obtaining target annotation results of the target object and mask annotation results of the target image, wherein the mask annotation results are used to characterize whether each image pixel in the target image belongs to the target object; constructing a first loss function based on the target annotation results and the target detection results; performing image recognition on the target image based on each feature map to obtain the mask detection results corresponding to each feature pixel in each feature map; constructing a second loss function based on the mask annotation results and the mask detection results; and adjusting the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0190] In the above embodiments of this application, constructing a first loss function based on target annotation results and target detection results includes: determining the intersection-union ratio (IU) of the annotation positions in the target annotation results and the detection positions in the target detection results to obtain the position IU; and constructing a first loss function based on the position IU and the score of the detection category in the target detection results.

[0191] In the above embodiments of this application, constructing a second loss function based on the mask annotation result and the mask detection result includes: scaling the mask annotation result based on the size of each feature map to obtain a sub-mask annotation result corresponding to each feature map; obtaining a sub-loss function corresponding to each feature map based on the sub-mask annotation result and the mask detection result corresponding to each feature map; and obtaining the sum of the sub-loss functions corresponding to multiple feature maps to obtain the second loss function.

[0192] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0193] Example 5

[0194] According to an embodiment of this application, an embodiment of an image detection method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order.

[0195] Figure 9 This is a flowchart of an image detection method according to an embodiment of the present invention, such as... Figure 9 As shown, the method may include the following steps:

[0196] Step S902: Upon receiving a communication request, acquire the target image.

[0197] The aforementioned communication request can be a video call request initiated by users. Users can initiate communication requests through mobile terminals such as smartphones, tablets, laptops, AR (Augmented Reality) devices, or VR (Virtual Reality) devices. After the communication request is initiated, the target detection results of the target object contained in the target image can be displayed in the call interface.

[0198] Step S904: Perform feature extraction on the target image multiple times to obtain multiple feature maps.

[0199] Among them, several feature maps have different resolutions.

[0200] Step S906: Perform feature fusion between each feature map and the corresponding historical feature map to obtain the target feature map.

[0201] Among them, the historical feature map is the feature map that has been fused before each feature map is extracted.

[0202] Step S908: Perform image detection on the target feature map to obtain the target detection result of the target object contained in the target image.

[0203] Step S910: Display the target detection results in the communication interface.

[0204] The communication interface in the above steps can be a video call interface, which can display the target detection results of two or more parties in the call.

[0205] For example, taking VR device communication as an example, when user A's VR device receives a video call request initiated by user B's VR device, if user A decides to have a video call with user B and needs to display the target detection result during the call, the VR device can be used to collect the target image and perform multiple feature extractions on the target image to obtain multiple feature maps. Each feature map is then fused with its corresponding historical feature map to obtain the target feature map. Image detection is then performed on the target feature map to obtain the target detection result of the target object contained in the target image. This target detection result can be displayed in the video call interface, which can be viewed by both user A and user B.

[0206] In the above embodiments of this application, performing multiple feature extractions on the target image to obtain multiple feature maps includes: inputting the target image into a feature extraction model and obtaining multiple feature maps output by the feature extraction model. The feature extraction model includes multiple feature extraction layers connected in sequence, each feature extraction layer being used to extract features from the input first image and output a feature map corresponding to the first image.

[0207] In the above embodiments of this application, feature fusion of each feature map with its corresponding historical feature map to obtain a target feature map includes: inputting multiple feature maps into a feature fusion model and obtaining the target feature map output by the feature fusion model. The feature fusion model includes: multiple feature fusion layers connected in sequence, with each feature fusion layer and multiple feature extraction layers connected in a one-to-one correspondence. Each feature fusion layer is connected to all other feature fusion layers. Each feature fusion layer is used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers and output the fused feature map.

[0208] In the above embodiments of this application, each feature fusion layer is used to perform attention operations on the first image and the feature map that has been fused to generate a fused feature map.

[0209] In the above embodiments of this application, image detection of the target feature map to obtain the target detection result of the target object contained in the target image includes: inputting the target feature map into the decoder and obtaining the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object and the second score of the target category; filtering the predicted position based on the first score and the second score to obtain the filtered predicted position; and generating the target detection result based on the filtered predicted position and the target category.

[0210] In the above embodiments of this application, filtering the predicted position based on the first score and the second score to obtain the filtered predicted position includes: obtaining the product of the first score and the second score to obtain the predicted score of the predicted position; and filtering the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0211] In the above embodiments of this application, the method further includes: obtaining target annotation results of the target object and mask annotation results of the target image, wherein the mask annotation results are used to characterize whether each image pixel in the target image belongs to the target object; constructing a first loss function based on the target annotation results and the target detection results; performing image recognition on the target image based on each feature map to obtain the mask detection results corresponding to each feature pixel in each feature map; constructing a second loss function based on the mask annotation results and the mask detection results; and adjusting the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0212] In the above embodiments of this application, constructing a first loss function based on target annotation results and target detection results includes: determining the intersection-union ratio (IU) of the annotation positions in the target annotation results and the detection positions in the target detection results to obtain the position IU; and constructing a first loss function based on the position IU and the score of the detection category in the target detection results.

[0213] In the above embodiments of this application, constructing a second loss function based on the mask annotation result and the mask detection result includes: scaling the mask annotation result based on the size of each feature map to obtain a sub-mask annotation result corresponding to each feature map; obtaining a sub-loss function corresponding to each feature map based on the sub-mask annotation result and the mask detection result corresponding to each feature map; and obtaining the sum of the sub-loss functions corresponding to multiple feature maps to obtain the second loss function.

[0214] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0215] Example 6

[0216] According to an embodiment of this application, an embodiment of an image detection method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order.

[0217] Figure 10 This is a flowchart of an image detection method according to an embodiment of the present invention, such as... Figure 10 As shown, the method may include the following steps:

[0218] Step S1002: Obtain the target image by calling the first interface.

[0219] The first interface includes a first parameter, the value of which is the target image.

[0220] The first interface in the above steps can be an interface for data interaction between the front-end client and the back-end server. The front-end client can pass the target image to the interface function as a parameter of the interface function to achieve the purpose of uploading the target image to the back-end server.

[0221] Step S1004: Perform feature extraction on the target image multiple times to obtain multiple feature maps.

[0222] Among them, several feature maps have different resolutions.

[0223] Step S1006: Perform feature fusion between each feature map and the historical feature map corresponding to each feature map to obtain the target feature map.

[0224] Among them, the historical feature map is the feature map that has been fused before each feature map is extracted.

[0225] Step S1008: Perform image detection on the target feature map to obtain the target detection result of the target object contained in the target image.

[0226] Step S1010: Output the target detection result by calling the second interface.

[0227] The second interface includes a second parameter, the value of which is the target detection result.

[0228] The second interface in the above steps can be an interface for data interaction between the backend server and the frontend client. The backend server can pass the target detection result to the interface function as a parameter of the interface function, so as to achieve the purpose of sending the target detection result to the frontend client.

[0229] In the above embodiments of this application, performing multiple feature extractions on the target image to obtain multiple feature maps includes: inputting the target image into a feature extraction model and obtaining multiple feature maps output by the feature extraction model. The feature extraction model includes multiple feature extraction layers connected in sequence, each feature extraction layer being used to extract features from the input first image and output a feature map corresponding to the first image.

[0230] In the above embodiments of this application, feature fusion of each feature map with its corresponding historical feature map to obtain a target feature map includes: inputting multiple feature maps into a feature fusion model and obtaining the target feature map output by the feature fusion model. The feature fusion model includes: multiple feature fusion layers connected in sequence, with each feature fusion layer and multiple feature extraction layers connected in a one-to-one correspondence. Each feature fusion layer is connected to all other feature fusion layers. Each feature fusion layer is used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers and output the fused feature map.

[0231] In the above embodiments of this application, each feature fusion layer is used to perform attention operations on the first image and the feature map that has been fused to generate a fused feature map.

[0232] In the above embodiments of this application, image detection of the target feature map to obtain the target detection result of the target object contained in the target image includes: inputting the target feature map into the decoder and obtaining the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object and the second score of the target category; filtering the predicted position based on the first score and the second score to obtain the filtered predicted position; and generating the target detection result based on the filtered predicted position and the target category.

[0233] In the above embodiments of this application, filtering the predicted position based on the first score and the second score to obtain the filtered predicted position includes: obtaining the product of the first score and the second score to obtain the predicted score of the predicted position; and filtering the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0234] In the above embodiments of this application, the method further includes: obtaining target annotation results of the target object and mask annotation results of the target image, wherein the mask annotation results are used to characterize whether each image pixel in the target image belongs to the target object; constructing a first loss function based on the target annotation results and the target detection results; performing image recognition on the target image based on each feature map to obtain the mask detection results corresponding to each feature pixel in each feature map; constructing a second loss function based on the mask annotation results and the mask detection results; and adjusting the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0235] In the above embodiments of this application, constructing a first loss function based on target annotation results and target detection results includes: determining the intersection-union ratio (IU) of the annotation positions in the target annotation results and the detection positions in the target detection results to obtain the position IU; and constructing a first loss function based on the position IU and the score of the detection category in the target detection results.

[0236] In the above embodiments of this application, constructing a second loss function based on the mask annotation result and the mask detection result includes: scaling the mask annotation result based on the size of each feature map to obtain a sub-mask annotation result corresponding to each feature map; obtaining a sub-loss function corresponding to each feature map based on the sub-mask annotation result and the mask detection result corresponding to each feature map; and obtaining the sum of the sub-loss functions corresponding to multiple feature maps to obtain the second loss function.

[0237] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0238] Example 7

[0239] According to embodiments of this application, an image detection apparatus for implementing the above-described image detection method is also provided, such as... Figure 11 As shown, the device 1100 includes: an acquisition module 1102, an extraction module 1104, a fusion module 1106, and a detection module 1108.

[0240] The system comprises the following modules: an acquisition module for acquiring the target image; an extraction module for performing multiple feature extractions on the target image to obtain multiple feature maps with different resolutions; a fusion module for fusing each feature map with its corresponding historical feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map was extracted; and a detection module for performing image detection on the target feature map to obtain the detection result of the target image.

[0241] It should be noted that the acquisition module 1102, extraction module 1104, fusion module 1106, and detection module 1108 mentioned above correspond to steps S202 to S208 in Embodiment 1. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 1.

[0242] In the above embodiments of this application, the extraction module is further used to input the target image into the feature extraction model and obtain multiple feature maps output by the feature extraction model. The feature extraction model includes multiple feature extraction layers connected in sequence. Each feature extraction layer is used to extract features from the input first image and output a feature map corresponding to the first image.

[0243] In the above embodiments of this application, the fusion module is further used to input multiple feature maps into the feature fusion model and obtain the target feature map output by the feature fusion model. The feature fusion model includes: multiple feature fusion layers connected in sequence, multiple feature fusion layers and multiple feature extraction layers are connected one-to-one, each feature fusion layer is connected to other feature fusion layers, and each feature fusion layer is used to fuse the input feature map with the feature maps that have been fused with other feature fusion layers, and output the fused feature map.

[0244] In the above embodiments of this application, the device is further configured to perform attention operations on the first image and the feature map that has been fused to generate a fused feature map in each feature fusion layer.

[0245] In the above embodiments of this application, the detection module includes: an input unit, a filtering unit, and a generation unit.

[0246] The input unit is used to input the target feature map into the decoder and obtain the initial detection result output by the decoder. The initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object, and the second score of the target category. The filtering unit is used to filter the predicted position based on the first score and the second score to obtain the filtered predicted position. The generation unit is used to generate the target detection result based on the filtered predicted position and the target category.

[0247] In the above embodiments of this application, the filtering unit is further configured to obtain the product of the first score and the second score to obtain the predicted score of the predicted position, and filter the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0248] In the above embodiments of this application, the device further includes: an identification module and an adjustment module.

[0249] The acquisition module is further used to acquire the target annotation results of the target object and the mask annotation results of the target image, wherein the mask annotation results are used to characterize whether each image pixel in the target image belongs to the target object; the construction module is further used to construct a first loss function based on the target annotation results and the target detection results; the recognition module is used to perform image recognition on the target image based on each feature map to obtain the mask detection results corresponding to each feature pixel in each feature map; the construction module is further used to construct a second loss function based on the mask annotation results and the mask detection results; and the adjustment module is used to adjust the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0250] In the above embodiments of this application, the construction module includes: a determination unit and a construction unit.

[0251] The determination unit is used to determine the intersection-union ratio (IUU) of the labeled positions in the target annotation results and the detection positions in the target detection results, thereby obtaining the position IUU; the construction unit is used to construct the first loss function based on the position IUU and the score of the detection category in the target detection results.

[0252] In the above embodiments of this application, the construction module further includes: a processing unit, a generation unit, and an acquisition unit.

[0253] The processing unit is used to construct a second loss function based on the mask annotation results and mask detection results, including: scaling the mask annotation results based on the size of each feature map to obtain a sub-mask annotation result corresponding to each feature map; the generation unit is used to obtain a sub-loss function corresponding to each feature map based on the sub-mask annotation results and the mask detection results corresponding to each feature map; and the acquisition unit is used to obtain the sum of the sub-loss functions corresponding to multiple feature maps to obtain the second loss function.

[0254] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0255] Example 8

[0256] According to embodiments of this application, an image detection apparatus for implementing the above-described image detection method is also provided, such as... Figure 12 As shown, the device 1200 includes: a receiving module 1202, an extraction module 1204, a fusion module 1206, a detection module 1208, and a sending module 1210.

[0257] The system comprises the following modules: a receiving module for receiving target images uploaded by clients via a cloud server; an extraction module for performing multiple feature extractions on the target image via a cloud server to obtain multiple feature maps with different resolutions; a fusion module for fusing each feature map with its corresponding historical feature map via a cloud server to obtain a target feature map, where the historical feature map is the feature map that was fused before each feature map was extracted; a detection module for performing image detection on the target feature map via a cloud server to obtain the target detection result of the target object contained in the target image; and a sending module for sending the target detection result to the client via a cloud server.

[0258] It should be noted that the receiving module 1202, extraction module 1204, fusion module 1206, detection module 1208, and sending module 1210 mentioned above correspond to steps S502 to S510 in Embodiment 2. The five modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0259] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0260] Example 9

[0261] According to embodiments of this application, an image detection apparatus for implementing the above-described image detection method is also provided, such as... Figure 13 As shown, the device 1300 includes: a receiving module 1302, an extraction module 1304, a fusion module 1306, a detection module 1308, and an output module 1310.

[0262] The system comprises: a receiving module for receiving the target image; an extraction module for performing multiple feature extractions on the target image to obtain multiple feature maps with different resolutions; a fusion module for fusing each feature map with its corresponding historical feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map was extracted; a detection module for performing image detection on the target feature map to obtain the target detection result of the target object contained in the target image; and an output module for outputting the target detection result.

[0263] It should be noted that the receiving module 1302, extraction module 1304, fusion module 1306, detection module 1308, and output module 1310 mentioned above correspond to steps S602 to S610 in Embodiment 3. The five modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0264] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0265] Example 10

[0266] According to embodiments of this application, an image detection apparatus for implementing the above-described image detection method is also provided, such as... Figure 14 As shown, the device 1400 includes: a display module 1402, a trigger module 1404, and a display module 1406.

[0267] The display module displays the target image on the interactive interface; the trigger module triggers multiple feature extractions of the target image to obtain multiple feature maps if a target detection command is detected in any area of ​​the interactive interface, and fuses each feature map with its corresponding historical feature map to obtain a target feature map. The multiple feature maps have different resolutions, and the historical feature maps are feature maps that have been fused before each feature map was extracted; the display module displays the target detection results of the target object contained in the target image on the interactive interface, where the target detection results are obtained by image detection of the target feature map.

[0268] It should be noted that the display module 1402, trigger module 1404, and display module 1406 mentioned above correspond to steps S702 to S706 in Embodiment 4. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0269] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0270] Example 11

[0271] According to embodiments of this application, an image detection apparatus for implementing the above-described image detection method is also provided, such as... Figure 15 As shown, the device 1500 includes: an acquisition module 1502, an extraction module 1504, a fusion module 1506, a detection module 1508, and a display module 1510.

[0272] The system comprises the following modules: an acquisition module for acquiring a target image upon receiving a communication request; an extraction module for performing multiple feature extractions on the target image to obtain multiple feature maps with different resolutions; a fusion module for fusing each feature map with its corresponding historical feature map to obtain a target feature map, wherein the historical feature map is the feature map that has already been fused before each feature map was extracted; a detection module for performing image detection on the target feature map to obtain the target detection result of the target object contained in the target image; and a display module for displaying the target detection result in the communication interface.

[0273] It should be noted that the acquisition module 1502, extraction module 1504, fusion module 1506, detection module 1508, and display module 1510 mentioned above correspond to steps S902 to S910 in Embodiment 5. The five modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 1.

[0274] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0275] Example 12

[0276] According to embodiments of this application, an image detection apparatus for implementing the above-described image detection method is also provided, such as... Figure 16 As shown, the device 1600 includes: a first calling module 1602, an extraction module 1604, a fusion module 1606, a detection module 1608, and a second calling module 1610.

[0277] The system comprises the following modules: a first calling module for acquiring a target image by calling a first interface, wherein the first interface includes a first parameter, the value of which is the target image; an extraction module for performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; a fusion module for fusing each feature map with its corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; a detection module for performing image detection on the target feature map to obtain a target detection result for the target object contained in the target image; and a second calling module for outputting the target detection result by calling a second interface, wherein the second interface includes a second parameter, the value of which is the target detection result.

[0278] It should be noted that the first calling module 1602, extraction module 1604, fusion module 1606, detection module 1608, and second calling module 1610 mentioned above correspond to steps S1002 to S1010 in Embodiment 6. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 1.

[0279] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0280] Example 13

[0281] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced by a mobile terminal or other terminal device.

[0282] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0283] In this embodiment, the computer terminal described above can execute the program code for the following steps in the image detection method: acquiring a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with the historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain the target detection result of the target object contained in the target image.

[0284] Optionally, Figure 17 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 17 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors and memory.

[0285] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image detection method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image detection method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0286] The processor can access information and application programs stored in the memory via a transmission device to perform the following steps: acquiring a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with its corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; and performing image detection on the target feature map to obtain the target detection result of the target object contained in the target image.

[0287] Optionally, the processor may also execute program code that performs the following steps: inputting the target image into the feature extraction model and obtaining multiple feature maps output by the feature extraction model, wherein the feature extraction model includes multiple feature extraction layers connected in sequence, each feature extraction layer being used to extract features from the input first image and output a feature map corresponding to the first image.

[0288] Optionally, the processor may also execute program code for the following steps: inputting multiple feature maps into a feature fusion model and obtaining the target feature map output by the feature fusion model, wherein the feature fusion model includes: multiple feature fusion layers connected in sequence, the multiple feature fusion layers and multiple feature extraction layers being connected one-to-one, each feature fusion layer being connected to other feature fusion layers, and each feature fusion layer being used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers, and outputting the fused feature map.

[0289] Optionally, the processor may also execute program code for the following steps: each feature fusion layer is used to perform attention operations on the first image and the feature map that has been fused, to generate a fused feature map.

[0290] Optionally, the processor may also execute program code that performs the following steps: inputting the target feature map into the decoder and obtaining the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object, and the second score of the target category; filtering the predicted position based on the first score and the second score to obtain the filtered predicted position; and generating the target detection result based on the filtered predicted position and the target category.

[0291] Optionally, the processor may also execute program code that performs the following steps: obtain the product of the first score and the second score to obtain the predicted score of the predicted position; filter the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0292] Optionally, the processor may also execute program code for the following steps: obtaining the target annotation result of the target object and the mask annotation result of the target image, wherein the mask annotation result is used to characterize whether each image pixel in the target image belongs to the target object; constructing a first loss function based on the target annotation result and the target detection result; performing image recognition on the target image based on each feature map to obtain the mask detection result corresponding to each feature pixel in each feature map; constructing a second loss function based on the mask annotation result and the mask detection result; and adjusting the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0293] Optionally, the processor may also execute program code that performs the following steps: determining the intersection-union ratio (IU) of the labeled positions in the target annotation results and the detection positions in the target detection results to obtain the position IU; and constructing a first loss function based on the position IU and the scores of the detection categories in the target detection results.

[0294] Optionally, the processor may also execute program code that performs the following steps: scaling the mask annotation results based on the size of each feature map to obtain sub-mask annotation results corresponding to each feature map; obtaining a sub-loss function corresponding to each feature map based on the sub-mask annotation results and the mask detection results corresponding to each feature map; and obtaining the sum of the sub-loss functions corresponding to multiple feature maps to obtain a second loss function.

[0295] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: the cloud server receives the target image uploaded by the client; the cloud server performs multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; the cloud server fuses each feature map with its corresponding historical feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map was extracted; the cloud server performs image detection on the target feature map to obtain the target detection result of the target object contained in the target image; the cloud server sends the target detection result to the client.

[0296] The processor can access information and application programs stored in the memory via a transmission device to perform the following steps: receiving a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with its corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain the target detection result of the target object contained in the target image; and outputting the target detection result.

[0297] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: displaying a target image on an interactive interface; if a target detection command is detected in any area of ​​the interactive interface, triggering multiple feature extractions of the target image to obtain multiple feature maps, and fusing each feature map with its corresponding historical feature map to obtain a target feature map, wherein the multiple feature maps have different resolutions, and the historical feature maps are feature maps that have been fused before each feature map was extracted; displaying the target detection result of the target object contained in the target image on the interactive interface, wherein the target detection result is obtained by image detection of the target feature map.

[0298] The processor can access information and applications stored in memory via a transmission device to perform the following steps: upon receiving a communication request, acquire a target image; perform multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fuse each feature map with its corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; perform image detection on the target feature map to obtain the target detection result of the target object contained in the target image; and display the target detection result in the communication interface.

[0299] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring a target image by calling a first interface, wherein the first interface includes a first parameter, the value of which is the target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with its corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain a target detection result for the target object contained in the target image; and outputting the target detection result by calling a second interface, wherein the second interface includes a second parameter, the value of which is the target detection result.

[0300] Those skilled in the art will understand that Figure 17 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 17 This does not limit the structure of the aforementioned electronic device. For example, computer terminal A may also include components that are more... Figure 17 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 17 The different configurations shown.

[0301] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0302] Example 14

[0303] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image detection method provided in the above embodiments.

[0304] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0305] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with the historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; and performing image detection on the target feature map to obtain the target detection result of the target object contained in the target image.

[0306] Optionally, the storage medium is further configured to store program code for performing the following steps: inputting a target image into a feature extraction model and obtaining multiple feature maps output by the feature extraction model, wherein the feature extraction model includes: multiple feature extraction layers connected in sequence, each feature extraction layer being used to extract features from the input first image and output a feature map corresponding to the first image.

[0307] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: inputting multiple feature maps into a feature fusion model and obtaining a target feature map output by the feature fusion model, wherein the feature fusion model includes: multiple feature fusion layers connected in sequence, the multiple feature fusion layers and multiple feature extraction layers being connected one-to-one, each feature fusion layer being connected to other feature fusion layers, each feature fusion layer being used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers, and outputting the fused feature map.

[0308] Optionally, the storage medium is further configured to store program code for performing the following steps: each feature fusion layer is used to perform attention operations on the first image and the feature map that has been fused to generate a fused feature map.

[0309] Optionally, the storage medium is further configured to store program code for performing the following steps: inputting the target feature map to the decoder and obtaining the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, a first score of the predicted position, the target category of the target object, and a second score of the target category; filtering the predicted position based on the first score and the second score to obtain the filtered predicted position; and generating the target detection result based on the filtered predicted position and the target category.

[0310] Optionally, the storage medium is further configured to store program code for performing the following steps: obtaining the product of the first score and the second score to obtain the predicted score of the predicted position; filtering the predicted position based on the predicted score and a preset threshold to obtain the filtered predicted position.

[0311] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: obtaining target annotation results of the target object and mask annotation results of the target image, wherein the mask annotation results are used to characterize whether each image pixel in the target image belongs to the target object; constructing a first loss function based on the target annotation results and the target detection results; performing image recognition on the target image based on each feature map to obtain the mask detection results corresponding to each feature pixel in each feature map; constructing a second loss function based on the mask annotation results and the mask detection results; and adjusting the model parameters of the feature extraction model, the feature fusion model, and the decoder based on the first loss function and the second loss function.

[0312] Optionally, the storage medium is further configured to store program code for performing the following steps: determining the intersection-union ratio (IU) of the labeled positions in the target annotation results and the detection positions in the target detection results to obtain the position IU; and constructing a first loss function based on the position IU and the score of the detection category in the target detection results.

[0313] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: scaling the mask annotation results based on the size of each feature map to obtain sub-mask annotation results corresponding to each feature map; obtaining a sub-loss function corresponding to each feature map based on the sub-mask annotation results and the mask detection results corresponding to each feature map; and obtaining the sum of the sub-loss functions corresponding to multiple feature maps to obtain a second loss function.

[0314] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the cloud server receives the target image uploaded by the client; the cloud server performs multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; the cloud server performs feature fusion between each feature map and the historical feature map corresponding to each feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map was extracted; the cloud server performs image detection on the target feature map to obtain the target detection result of the target object contained in the target image; the cloud server sends the target detection result to the client.

[0315] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with the historical feature map corresponding to each feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain a target detection result of the target object contained in the target image; and outputting the target detection result.

[0316] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: displaying a target image on an interactive interface; if a target detection command is detected in any area of ​​the interactive interface, triggering multiple feature extractions of the target image to obtain multiple feature maps, and fusing each feature map with the corresponding historical feature map to obtain a target feature map, wherein the multiple feature maps have different resolutions, and the historical feature map is a feature map that has been fused before each feature map was extracted; displaying the target detection result of the target object contained in the target image on the interactive interface, wherein the target detection result is obtained by performing image detection on the target feature map.

[0317] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: upon receiving a communication request, acquiring a target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with the corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain the target detection result of the target object contained in the target image; and displaying the target detection result in the communication interface.

[0318] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a target image by calling a first interface, wherein the first interface includes: a first parameter, the parameter value of the first parameter being the target image; performing multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; fusing each feature map with a corresponding historical feature map to obtain a target feature map, wherein the historical feature map is a feature map that has been fused before each feature map was extracted; performing image detection on the target feature map to obtain a target detection result of the target object contained in the target image; and outputting the target detection result by calling a second interface, wherein the second interface includes: a second parameter, the parameter value of the second parameter being the target detection result.

[0319] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0320] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0321] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0322] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0323] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0324] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0325] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image detection method, characterized in that, include: Acquire the target image; Multiple feature extractions are performed on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; Each feature map is fused with the corresponding historical feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map is extracted; The target feature map is input to the decoder, and the initial detection result output by the decoder is obtained. The initial detection result includes: the predicted location of the target object, the first score of the predicted location, the target category of the target object, and the second score of the target category. The predicted position is filtered based on the first score and the second score to obtain the filtered predicted position; Based on the filtered predicted location and the target category, a target detection result is generated.

2. The method according to claim 1, characterized in that, Multiple feature extractions are performed on the target image to obtain multiple feature maps, including: The target image is input into a feature extraction model, and the multiple feature maps output by the feature extraction model are obtained. The feature extraction model includes multiple feature extraction layers connected in sequence. Each feature extraction layer is used to extract features from the input first image and output a feature map corresponding to the first image.

3. The method according to claim 2, characterized in that, Each feature map is fused with its corresponding historical feature map to obtain the target feature map, which includes: The multiple feature maps are input into a feature fusion model, and the target feature map output by the feature fusion model is obtained. The feature fusion model includes multiple feature fusion layers connected in sequence. The multiple feature fusion layers and the multiple feature extraction layers are connected one-to-one. Each feature fusion layer is connected to other feature fusion layers. Each feature fusion layer is used to fuse the input feature map with the feature maps that have been fused by other feature fusion layers, and output the fused feature map.

4. The method according to claim 3, characterized in that, Each feature fusion layer is used to perform attention operations on the first image and the fused feature map to generate the fused feature map.

5. The method according to claim 1, characterized in that, The predicted positions are filtered based on the first score and the second score to obtain the following filtered predicted positions: The product of the first score and the second score is obtained to get the predicted score of the predicted position; The predicted position is filtered based on the predicted score and a preset threshold to obtain the filtered predicted position.

6. The method according to any one of claims 2 to 4, characterized in that, The method further includes: Obtain the target annotation result of the target object and the mask annotation result of the target image, wherein the mask annotation result is used to characterize whether each image pixel in the target image belongs to the target object; Based on the target annotation results and the target detection results, a first loss function is constructed; Image recognition is performed on the target image based on each feature map to obtain the mask detection result corresponding to each feature pixel in each feature map; Based on the mask annotation results and the mask detection results, a second loss function is constructed; The model parameters of the feature extraction model, feature fusion model, and decoder are adjusted based on the first loss function and the second loss function.

7. The method according to claim 6, characterized in that, Based on the target annotation results and the target detection results, the first loss function is constructed as follows: The intersection-union ratio (IUU) of the labeled positions in the target annotation results and the detected positions in the target detection results is determined to obtain the position IUU. The first loss function is constructed based on the intersection-union ratio at the location and the score of the detection category in the target detection result.

8. The method according to claim 6, characterized in that, Based on the mask annotation results and the mask detection results, the second loss function is constructed as follows: Based on the size of each feature map, the mask annotation result is scaled to obtain the sub-mask annotation result corresponding to each feature map; Based on the sub-mask annotation results and the mask detection results corresponding to each feature map, the sub-loss function corresponding to each feature map is obtained; The second loss function is obtained by summing the sub-loss functions corresponding to the multiple feature maps.

9. An image detection method, characterized in that, include: The cloud server receives the target image uploaded by the client; The cloud server performs multiple feature extractions on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; The cloud server performs feature fusion between each feature map and the corresponding historical feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map is extracted; The cloud server inputs the target feature map into the decoder and obtains the initial detection result output by the decoder, wherein the initial detection result includes: the predicted position of the target object, the first score of the predicted position, the target category of the target object, and the second score of the target category; The cloud server filters the predicted location based on the first score and the second score to obtain the filtered predicted location; The cloud server generates target detection results based on the filtered predicted location and the target category; The cloud server sends the target detection result to the client.

10. An image detection method, characterized in that, include: Receive the target image; Multiple feature extractions are performed on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; Each feature map is fused with the corresponding historical feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map is extracted; The target feature map is input to the decoder, and the initial detection result output by the decoder is obtained. The initial detection result includes: the predicted location of the target object, the first score of the predicted location, the target category of the target object, and the second score of the target category. The predicted position is filtered based on the first score and the second score to obtain the filtered predicted position; Based on the filtered predicted location and the target category, a target detection result is generated; Output the target detection results.

11. An image detection method, characterized in that, include: Display the target image on the interactive interface; If a target detection command is detected in any area of ​​the interactive interface, multiple feature extractions are triggered on the target image to obtain multiple feature maps. Each feature map is then fused with the corresponding historical feature map to obtain a target feature map. The multiple feature maps have different resolutions, and the historical feature map is a feature map that has been fused before each feature map was extracted. The target detection results of the target objects contained in the target image are displayed on the interactive interface. The target detection results are generated based on the predicted position of the target objects contained in the target image and the target category corresponding to the target objects. The predicted position is obtained by filtering the initial predicted position based on the first score of the initial predicted position obtained by inputting the target feature map into the decoder and the second score of the target category. The target category is obtained based on the target feature map input into the decoder.

12. An image detection method, characterized in that, include: Upon receiving a communication request, acquire the target image; Multiple feature extractions are performed on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; Each feature map is fused with the corresponding historical feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map is extracted; The target feature map is input to the decoder, and the initial detection result output by the decoder is obtained. The initial detection result includes: the predicted location of the target object, the first score of the predicted location, the target category of the target object, and the second score of the target category. The predicted position is filtered based on the first score and the second score to obtain the filtered predicted position; Based on the filtered predicted location and the target category, a target detection result is generated; The target detection results are displayed in the communication interface.

13. An image detection method, characterized in that, include: The target image is obtained by calling a first interface, wherein the first interface includes: a first parameter, the value of which is the target image; Multiple feature extractions are performed on the target image to obtain multiple feature maps, wherein the multiple feature maps have different resolutions; Each feature map is fused with the corresponding historical feature map to obtain the target feature map, wherein the historical feature map is the feature map that has been fused before each feature map is extracted; The target feature map is input to the decoder, and the initial detection result output by the decoder is obtained. The initial detection result includes: the predicted location of the target object, the first score of the predicted location, the target category of the target object, and the second score of the target category. The predicted position is filtered based on the first score and the second score to obtain the filtered predicted position; Based on the filtered predicted location and the target category, a target detection result is generated; The target detection result is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target detection result.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the image detection method according to any one of claims 1 to 13.

15. A computer terminal, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when executed, performs the image detection method according to any one of claims 1 to 13.

Images