Object detection method and related device thereof

Abstract

Disclosed in the present application are an object detection method and a related device thereof, which can quickly complete object detection tasks, thus improving the efficiency of completing object detection tasks, and furthermore improving user experience. The method of the present application comprises: when object detection needs to be performed on an object image, first acquiring the object image containing an object to be detected, and inputting the object image into an object model; next, the object model can perform feature extraction on the object image to obtain a first feature of the object image; then, the object model can encode the first feature of the object image to obtain a second feature of the object image; then, on the basis of a preset query vector, the object model can decode the second feature of the object image to obtain a third feature of the object image; and finally, the object model can acquire a detection result of the object image on the basis of the third feature, wherein the detection result can be used for determining position information of the object and the category of the object. Thus, the object detection of the object model is completed.

Description

A target detection method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 29, 2023, with application number 202310792993.X and invention name “A target detection method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to artificial intelligence (AI), and more particularly to a method for detecting a target and related equipment. Background Art

[0003] Object detection is one of the most fundamental computer vision tasks and is crucial for many practical applications. In recent years, the Transformer model and its variants have demonstrated outstanding performance in image classification tasks, so they can be adapted to handle object detection tasks to improve their performance.

[0004] Currently, when it is necessary to detect the objects contained in the target image, the target image can be first input into the transformer model. Then, the transformer model can first extract the initial features of the target image. Next, the transformer model can process the initial features of the target image based on the self-attention mechanism to obtain the intermediate features of the target image. Then, the transformer model can process the intermediate features of the target image based on the self-attention mechanism and the cross-attention mechanism to obtain the final features of the target image. Finally, the transformer model can obtain the detection results of the target image based on the final features of the target image, and thus determine the location information and category of the object.

[0005] In the above process, since the transformer model is mainly built based on the attention mechanism, it will consume a lot of computing costs when performing target detection. If the computing power of the device equipped with the transformer model is low, the target detection speed will be slow, affecting the efficiency of target detection, and thus resulting in a poor user experience.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a target detection method and related equipment, which can quickly complete target detection tasks, improve the efficiency of completing target detection tasks, and thus improve user experience.

[0008] A first aspect of an embodiment of the present application provides a target detection method, which can be implemented using a target model. The method includes:

[0009] When target detection is required for a target image, the target image and a preset query vector may be obtained first, wherein the target image contains one or more objects to be detected.

[0010] After obtaining the target image and query vector, the target image and the preset query vector can be input into the target model. The target model can then first perform feature extraction on the target image to obtain the first feature of the target image. After obtaining the first feature of the target image, the target model can encode the first feature of the target image to obtain the second feature of the target image. After obtaining the second feature of the target image, the target model can use the preset query vector to decode the second feature of the target image to obtain the third feature of the target image. After obtaining the third feature of the target image, the target model can further process the third feature of the target image to obtain the detection result of the target image.

[0011] It is worth noting that the encoding performed by the target model on the first feature of the target image may include at least one convolution, but does not include any attention-based processing. Furthermore, the decoding performed by the target model on the second feature of the target image may include at least one convolution, but also does not include any attention-based processing.

[0012] The detection result of the target image includes the location information of at least one object detected by the model, the category of the at least one object, and the confidence level of the at least one object. Therefore, the location information and category of objects with low confidence levels are discarded, and the location information and category of objects with high confidence levels are retained as the final location information and category of the objects detected by the model. At this point, object detection for the target image is completed.

[0013] As can be seen from the above method, when object detection is required for a target image, a target image containing the object to be detected can be first obtained and input into the target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location and category. This completes object detection for the target model. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations involve at least one convolution and do not include any attention-based processing. This reduces the computational cost of the target model during object detection. Even if the computing power of the device hosting the target model is low, the target detection task can be completed quickly, improving efficiency and ultimately enhancing the user experience.

[0014] In one possible implementation, the encoding includes at least one of: channel-by-channel convolution or point-by-point convolution. In the aforementioned implementation, the target model may perform at least one channel-by-channel convolution and / or at least one point-by-point convolution on the first feature of the target image to obtain the second feature of the target image.

[0015] In one possible implementation, decoding the second feature based on a preset query vector to obtain the third feature of the target image includes: performing a first processing on the preset query vector to obtain a fourth feature of the query vector, the first processing including channel-by-channel convolution and point-by-point convolution; performing a second processing on the second and fourth features to obtain a fifth feature of the target image, the second processing including channel-by-channel convolution; and performing a third processing on the fifth feature to obtain the third feature of the target image. In the aforementioned implementation, after obtaining the preset query vector and the second feature of the target image, the target model may first perform the first processing on the preset query vector to obtain the fourth feature of the query vector. The first processing performed by the target model on the query vector may include at least channel-by-channel convolution and point-by-point convolution, but may not include any processing based on an attention mechanism. After obtaining the fourth feature of the query vector, the target model may perform a second processing on the second feature of the target image and the fourth feature of the query vector to obtain the fifth feature of the target image. The second processing performed by the target model on the second and fourth features may include at least channel-by-channel convolution, but may not include any processing based on an attention mechanism. After obtaining the fifth feature of the target image, the target model may perform a third processing on the fifth feature of the target image, thereby obtaining the third feature of the target image.

[0016] In one possible implementation, performing a first processing on a preset query vector to obtain a fourth feature of the query vector includes: performing channel-by-channel convolution and point-by-point convolution on the preset query vector to obtain a sixth feature of the query vector; and adding the sixth feature to the query vector to obtain the fourth feature of the query vector. In the aforementioned implementation, after obtaining the preset query vector, the target model may first perform at least one channel-by-channel convolution and at least one point-by-point convolution on the preset query vector to obtain the sixth feature of the query vector. After obtaining the sixth feature of the query vector, the target model may further add the sixth feature of the query vector to the query vector to obtain the fourth feature of the query vector.

[0017] In one possible implementation, performing a second processing on the second and fourth features to obtain the fifth feature of the target image includes: upsampling the fourth feature to obtain the seventh feature of the query vector; fusing the second and seventh features to obtain the eighth feature of the target image; performing a channel-by-channel convolution on the eighth feature to obtain the ninth feature of the target image; and adding the ninth feature to the seventh feature to obtain the fifth feature of the target image. In the aforementioned implementation, after obtaining the fourth feature of the query vector, the target model may first upsample the fourth feature of the query vector to obtain the seventh feature of the query vector. After obtaining the seventh feature of the query vector and the second feature of the target image, the target model may fuse the second feature of the target image and the seventh feature of the query vector to obtain the eighth feature of the target image. After obtaining the eighth feature of the target image, the target model may perform at least one channel-by-channel convolution on the eighth feature of the target image to obtain the ninth feature of the target image. After obtaining the ninth feature of the target image, the target model may add the ninth feature of the target image and the seventh feature of the query vector to obtain the fifth feature of the target image.

[0018] In one possible implementation, performing a third processing on the fifth feature to obtain the third feature of the target image includes: performing feedforward neural network-based processing on the fifth feature to obtain the tenth feature of the target image; adding the fifth feature and the tenth feature to obtain the eleventh feature of the target image; and pooling the eleventh feature to obtain the third feature of the target image. In the aforementioned implementation, after obtaining the fifth feature of the target image, the target model may first perform feedforward neural network-based processing on the fifth feature to obtain the tenth feature of the target image. After obtaining the tenth feature of the target image, the target model may add the fifth feature of the target image and the tenth feature of the target image to obtain the eleventh feature of the target image. After obtaining the eleventh feature of the target image, the target model may pool the eleventh feature of the target image to obtain the third feature of the target image.

[0019] In one possible implementation, the channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

[0020] In one possible implementation, the point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

[0021] In one possible implementation, the query vector includes multiple parameters, and the number of parameters is related to the number of objects.

[0022] A second aspect of an embodiment of the present application provides a model training method, which includes: obtaining a training image, where the training image contains an object to be detected; processing the training image through the model to be trained to obtain a detection result of the training image, where the detection result is used to determine the location information of the object and the category of the object, and the model to be trained is used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, where the encoding includes at least one convolution and does not include processing based on an attention mechanism; decode the second feature based on a preset query vector to obtain a third feature of the training image, where the decoding includes at least one convolution and does not include processing based on an attention mechanism; obtain a detection result of the training image based on the third feature; train the model to be trained based on the detection result and the actual detection result of the training image to obtain a target model.

[0023] The target model trained using the above method has object detection capabilities. Specifically, when target detection is required for a target image, a target image containing the object to be detected can be first obtained and input into the target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location and category. This completes target detection for the target model. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This reduces the computational cost of the target model during target detection. Even if the computing power of the device hosting the target model is low, the target detection task can be completed quickly, improving efficiency and ultimately enhancing the user experience.

[0024] In one possible implementation, the encoding includes at least one of the following: channel-by-channel convolution or point-by-point convolution.

[0025] In one possible implementation, decoding the second feature based on a preset query vector to obtain the third feature of the training image includes: performing a first processing on the preset query vector to obtain a fourth feature of the query vector, the first processing including channel-by-channel convolution and point-by-point convolution; performing a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, the second processing including channel-by-channel convolution; and performing a third processing on the fifth feature to obtain the third feature of the training image.

[0026] In one possible implementation, performing a first processing on a preset query vector to obtain a fourth feature of the query vector includes: performing channel-by-channel convolution and point-by-point convolution on the preset query vector to obtain a sixth feature of the query vector; and adding the sixth feature and the query vector to obtain a fourth feature of the query vector.

[0027] In one possible implementation, performing a second processing on the second feature and the fourth feature to obtain the fifth feature of the training image includes: upsampling the fourth feature to obtain the seventh feature of the query vector; fusing the second feature and the seventh feature to obtain the eighth feature of the training image; performing channel-by-channel convolution on the eighth feature to obtain the ninth feature of the training image; and adding the ninth feature and the seventh feature to obtain the fifth feature of the training image.

[0028] In one possible implementation, performing a third processing on the fifth feature to obtain the third feature of the training image includes: performing a feedforward neural network-based processing on the fifth feature to obtain the tenth feature of the training image; adding the fifth feature and the tenth feature to obtain the eleventh feature of the training image; and pooling the eleventh feature to obtain the third feature of the training image.

[0029] In one possible implementation, the channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

[0030] In one possible implementation, the point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

[0031] In one possible implementation, the query vector includes multiple parameters, and the number of parameters is related to the number of objects.

[0032] A third aspect of an embodiment of the present application provides a target detection device, which includes a target model and comprises: an acquisition module for acquiring a target image, wherein the target image includes an object to be detected; an extraction module for performing feature extraction on the target image to obtain a first feature of the target image; an encoding module for encoding the first feature to obtain a second feature of the target image, wherein the encoding includes at least one convolution and does not include processing based on an attention mechanism; a decoding module for decoding the second feature based on a preset query vector to obtain a third feature of the target image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism; a detection module for obtaining a detection result of the target image based on the third feature, wherein the detection result is used to determine the location information of the object and the category of the object.

[0033] As can be seen from the above device, when target detection is required for a target image, a target image containing the object to be detected can be first obtained and input into the target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location and category. This completes target detection for the target model. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This reduces the computational cost of the target model during target detection. Even if the computing power of the device hosting the target model is low, the target detection task can be completed quickly, improving efficiency and ultimately enhancing the user experience.

[0034] In one possible implementation, the encoding includes at least one of the following: channel-by-channel convolution or point-by-point convolution.

[0035] In one possible implementation, the decoding module is used to: perform a first processing on a preset query vector to obtain a fourth feature of the query vector, where the first processing includes channel-by-channel convolution and point-by-point convolution; perform a second processing on the second feature and the fourth feature to obtain a fifth feature of the target image, where the second processing includes channel-by-channel convolution; and perform a third processing on the fifth feature to obtain a third feature of the target image.

[0036] In one possible implementation, the decoding module is configured to: perform channel-by-channel convolution and point-by-point convolution on a preset query vector to obtain a sixth feature of the query vector; and add the sixth feature and the query vector to obtain a fourth feature of the query vector.

[0037] In one possible implementation, the decoding module is used to: upsample the fourth feature to obtain the seventh feature of the query vector; fuse the second feature and the seventh feature to obtain the eighth feature of the target image; perform channel-by-channel convolution on the eighth feature to obtain the ninth feature of the target image; and add the ninth feature and the seventh feature to obtain the fifth feature of the target image.

[0038] In one possible implementation, the decoding module is used to: process the fifth feature based on a feedforward neural network to obtain a tenth feature of the target image; add the fifth feature and the tenth feature to obtain an eleventh feature of the target image; and pool the eleventh feature to obtain a third feature of the target image.

[0039] In one possible implementation, the channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

[0040] In one possible implementation, the point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

[0041] In one possible implementation, the query vector includes multiple parameters, and the number of parameters is related to the number of objects.

[0042] The fourth aspect of an embodiment of the present application provides a model training device, which includes: an acquisition module for acquiring a training image, where the training image contains an object to be detected; a processing module for processing the training image through the model to be trained to obtain a detection result of the training image, where the detection result is used to determine the position information of the object and the category of the object, and the model to be trained is used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, where the encoding includes at least one convolution and does not include processing based on an attention mechanism; decode the second feature based on a preset query vector to obtain a third feature of the training image, where the decoding includes at least one convolution and does not include processing based on an attention mechanism; obtain the detection result of the training image based on the third feature; a training module for training the model to be trained based on the detection result and the actual detection result of the training image to obtain a target model.

[0043] The target model trained in the embodiments of the present application has target detection capabilities. Specifically, when target detection is required for a target image, a target image containing the object to be detected can be first obtained and input into the target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location information and category. At this point, target detection for the target model is complete. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection. Even if the computing power of the device equipped with the target model is low, the target detection task can be completed quickly, improving the efficiency of the target detection task and, in turn, enhancing the user experience.

[0044] In one possible implementation, the encoding includes at least one of the following: channel-by-channel convolution or point-by-point convolution.

[0045] In one possible implementation, the model to be trained is used to: perform a first processing on a preset query vector to obtain a fourth feature of the query vector, where the first processing includes channel-by-channel convolution and point-by-point convolution; perform a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, where the second processing includes channel-by-channel convolution; and perform a third processing on the fifth feature to obtain a third feature of the training image.

[0046] In one possible implementation, the model to be trained is used to: perform channel-by-channel convolution and point-by-point convolution on a preset query vector to obtain a sixth feature of the query vector; and add the sixth feature and the query vector to obtain a fourth feature of the query vector.

[0047] In one possible implementation, the model to be trained is used to: upsample the fourth feature to obtain the seventh feature of the query vector; fuse the second feature and the seventh feature to obtain the eighth feature of the training image; perform channel-by-channel convolution on the eighth feature to obtain the ninth feature of the training image; and add the ninth feature and the seventh feature to obtain the fifth feature of the training image.

[0048] In one possible implementation, the model to be trained is used to: process the fifth feature based on a feedforward neural network to obtain the tenth feature of the training image; add the fifth feature and the tenth feature to obtain the eleventh feature of the training image; and pool the eleventh feature to obtain the third feature of the training image.

[0049] In one possible implementation, the channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

[0050] In one possible implementation, the point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

[0051] In one possible implementation, the query vector includes multiple parameters, and the number of parameters is related to the number of objects.

[0052] A fifth aspect of an embodiment of the present application provides a target detection device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the target detection device executes the method described in the first aspect or any possible implementation method of the first aspect.

[0053] A sixth aspect of an embodiment of the present application provides a model training device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes the method described in the second aspect or any possible implementation method of the second aspect.

[0054] A seventh aspect of an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0055] An eighth aspect of an embodiment of the present application provides a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes a method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0056] In a possible implementation, the processor is coupled to the memory through an interface.

[0057] In one possible implementation, the chip system further includes a memory in which computer programs or computer instructions are stored.

[0058] A ninth aspect of an embodiment of the present application provides a computer storage medium storing a computer program, which, when executed by a computer, enables the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0059] A tenth aspect of an embodiment of the present application provides a computer program product, which stores instructions that, when executed by a computer, enable the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0060] In an embodiment of the present application, when target detection is required for a target image, a target image containing the object to be detected can be first obtained and input into a target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location information and category. At this point, target detection for the target model is complete. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection. Even if the computing power of the device equipped with the target model is low, the target detection task can be completed quickly, improving the efficiency of the target detection task and thereby enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;

[0062] FIG2a is a schematic structural diagram of a target detection system provided in an embodiment of the present application;

[0063] FIG2 b is another schematic diagram of the structure of the target detection system provided in an embodiment of the present application;

[0064] FIG2c is a schematic diagram of a target detection device according to an embodiment of the present application;

[0065] FIG3 is a schematic diagram of the architecture of the system 100 provided in an embodiment of the present application;

[0066] FIG4 is a schematic diagram of a structure of a target model provided in an embodiment of the present application;

[0067] FIG5 is a flow chart of a target detection method provided in an embodiment of the present application;

[0068] FIG6 is a schematic diagram of a structure of a target model provided in an embodiment of the present application;

[0069] FIG7 is a schematic structural diagram of a decoder provided in an embodiment of the present application;

[0070] FIG8 is another schematic diagram of the structure of the target model provided in an embodiment of the present application;

[0071] FIG9 is a schematic diagram of a comparison result provided by an embodiment of the present application;

[0072] FIG10 is a flow chart of a model training method provided in an embodiment of the present application;

[0073] FIG11 is a schematic structural diagram of a target detection device provided in an embodiment of the present application;

[0074] FIG12 is a schematic structural diagram of a model training device provided in an embodiment of the present application;

[0075] FIG13 is a schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0076] FIG14 is a schematic structural diagram of a training device provided in an embodiment of the present application;

[0077] FIG15 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The embodiments of the present application provide a target detection method and related equipment, which can quickly complete target detection tasks, improve the efficiency of completing target detection tasks, and thus improve user experience.

[0079] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0080] Object detection is one of the most fundamental computer vision tasks and is crucial for many practical applications. In recent years, the Transformer model and its variants have demonstrated outstanding performance in image classification tasks, so they can be adapted to handle object detection tasks to improve their performance.

[0081] Currently, when it is necessary to detect objects contained in a target image, the target image can be input into a transformer model. The transformer model then extracts the initial features of the target image. Next, the transformer model processes these initial features using a self-attention mechanism to obtain intermediate features of the target image. The transformer model then processes these intermediate features using a self-attention mechanism and a cross-attention mechanism to obtain final features of the target image. Finally, the transformer model obtains detection results based on the final features of the target image, thereby determining the location and category of the objects. For example, suppose an image contains multiple objects such as a cat, grass, flowers, and trees. This image can be input into a transformer model, which processes the image to obtain detection results. Based on these detection results, the locations of the multiple objects in the image and the categories to which they belong can be determined.

[0082] In the above process, since the transformer model is mainly built based on the attention mechanism, it will consume a lot of computing costs when performing target detection. If the computing power of the device equipped with the transformer model is low, the target detection speed will be slow, affecting the efficiency of target detection, and thus resulting in a poor user experience.

[0083] In order to solve the above problems, an embodiment of the present application provides a target detection method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by perceiving the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for data processing is a common application of artificial intelligence.

[0084] First, we will describe the overall workflow of an AI system. See Figure 1, which illustrates a schematic diagram of the AI ​​framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0085] (1) Infrastructure

[0086] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0087] (2) Data

[0088] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0089] (3) Data processing

[0090] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0091] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0092] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0093] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0094] (4) General ability

[0095] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0096] (5) Smart products and industry applications

[0097] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0098] Next, several application scenarios of this application are introduced.

[0099] Figure 2a is a schematic diagram of the structure of a target detection system provided in an embodiment of the present application. The target detection system includes a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, a personal computer, or an information processing center. The user device is the initiator of target detection and is the initiator of the target detection request, typically initiated by a user through the user device.

[0100] The aforementioned data processing device can be a device or server with data processing capabilities, such as a cloud server, network server, application server, or management server. The data processing device receives target detection requests from smart terminals via an interactive interface. It then uses its data storage and data processing processor to perform target detection processing using methods such as machine learning, deep learning, search, reasoning, and decision-making. The memory in the data processing device is a general term that includes local storage and a database that stores historical data. The database can be located on the data processing device or on other network servers.

[0101] In the target detection system shown in FIG2a, the user device can receive user instructions. For example, the user device can obtain an image input / selected by the user, and then initiate a request to the data processing device, causing the data processing device to perform target detection processing on the image on the user device, thereby obtaining a target detection result for the image, and then determining the position information of several objects in the image and the categories of these objects. For example, the user can input or select a target image containing an object to be detected on the user device, and then the user device initiates a target detection request for the target image to the data processing device, causing the data processing device to perform target detection processing on the target image, thereby obtaining a target detection result for the target image. Therefore, the position information of the object to be detected and the category of the object can be determined based on the target detection result.

[0102] In FIG. 2 a , the data processing device may execute the target detection method according to the embodiment of the present application.

[0103] Figure 2b is another structural schematic diagram of the target detection system provided in an embodiment of the present application. In Figure 2b, the user device directly serves as a data processing device. After determining a certain image input / selected by the user, the user device itself can directly perform target detection processing on the image. The specific process is similar to that of Figure 2a. Please refer to the above description and will not be repeated here.

[0104] In the target detection system shown in FIG2b , the user device can receive user instructions. For example, the user can input into the user device or select a target image containing an object to be detected on the user device. The user device then performs target detection processing on the target image to obtain a target detection result of the target image. Therefore, the location information of the object to be detected and the category of the object can be determined based on the target detection result.

[0105] In FIG2 b , the user equipment itself can execute the target detection method of the embodiment of the present application.

[0106] FIG2c is a schematic diagram of a target detection device according to an embodiment of the present application.

[0107] The user device in the above Figures 2a and 2b can specifically be the local device 301 or the local device 302 in Figure 2c, and the data processing device in Figure 2a can specifically be the execution device 210 in Figure 2c, wherein the data storage system 250 can store the data to be processed of the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or it can be set on the cloud or other network servers.

[0108] The processors in Figures 2a and 2b can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to perform target detection processing on the image, thereby obtaining corresponding target detection results.

[0109] Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of the present application. In Figure 3, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data can include: various tasks to be scheduled, callable resources and other parameters in the embodiment of the present application.

[0110] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing (such as implementing the functions of the neural network in this application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0111] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0112] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from training samples collected by the data collection device 160.

[0113] In the scenario shown in FIG3 , the user can manually input data, which can be performed through the interface provided by I / O interface 112. Alternatively, client device 140 can automatically send input data to I / O interface 112. If user authorization is required for client device 140 to automatically send input data, the user can set the corresponding permissions in client device 140. The user can view the output of execution device 110 on client device 140, which can be presented in a display, sound, action, or other specific form. Client device 140 can also serve as a data acquisition terminal, collecting input data and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, collection can also be performed without client device 140, with I / O interface 112 directly storing the input data and output results from I / O interface 112 as new sample data in database 130.

[0114] It is worth noting that FIG3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG3, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110. As shown in FIG3, a neural network can be trained according to the training device 120.

[0115] The present application also provides a chip including a neural network processor (NPU). The chip can be provided in the execution device 110 shown in FIG3 to complete the computational work of the computation module 111. The chip can also be provided in the training device 120 shown in FIG3 to complete the training work of the training device 120 and output the target model / rule.

[0116] The neural network processor (NPU) is a coprocessor mounted on the host central processing unit (CPU), which assigns tasks to it. The core of the NPU is the arithmetic circuit, which is controlled by a controller to extract data from memory (weight memory or input memory) and perform calculations.

[0117] In some implementations, the arithmetic circuit includes multiple processing engines (PEs). In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.

[0118] For example, consider an input matrix A, a weight matrix B, and an output matrix C. The computational circuit retrieves the corresponding data for matrix B from the weight memory and caches it on each PE within the computational circuit. The computational circuit then performs a matrix operation on the matrix A data from the input memory and matrix B, storing the partial or final matrix results in the accumulator.

[0119] The vector computation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit can be used for network calculations in non-convolutional / non-FC layers in neural networks, such as pooling, batch normalization, and local response normalization.

[0120] In some implementations, the vector computation unit can store the processed output vectors in a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the computation circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vectors can be used as activation inputs to the computation circuit, such as for use in subsequent layers in a neural network.

[0121] The unified memory is used to store input data and output data.

[0122] The weight data is directly transferred from the external memory to the input memory and / or unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.

[0123] The bus interface unit (BIU) is used to implement interaction between the main CPU, DMAC and instruction fetch memory through the bus.

[0124] An instruction fetch buffer connected to the controller, used to store instructions used by the controller;

[0125] The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.

[0126] Generally, unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and external memory is memory outside the NPU, which can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.

[0127] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0128] (1) Neural Network

[0129] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0130] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0131] The operation of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the operation of each layer in a neural network can be understood as transforming the input space (a set of input vectors) into the output space (i.e., from the row space to the column space of a matrix) through five operations. These operations include: 1. Dimensionality increase / decrease; 2. Scaling / reduction; 3. Rotation; 4. Translation; and 5. "Bending." Operations 1, 2, and 3 are performed by Wx, 4 by +b, and 5 by a(). The word "space" is used here because the objects being classified are not individual things, but rather a class of things, and space refers to the collection of all individuals within that class. W is the weight vector, each value in which represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from input space to output space described above. That is, the weights W of each layer control how the space is transformed. The goal of training a neural network is to ultimately obtain the weight matrix for all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, and more specifically, about learning the weight matrix.

[0132] Because we want the output of the neural network to be as close as possible to the value we actually want to predict, we compare the current network's predicted value with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value, and this adjustment is continued until the neural network can predict the desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0133] (2) Backpropagation algorithm

[0134] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0135] The method provided in this application is described below from the training side of the neural network and the application side of the neural network.

[0136] The model training method provided in the embodiment of the present application involves the processing of data sequences, and can be specifically applied to methods such as data training, machine learning, and deep learning, and performs symbolic and formalized intelligent information modeling, extraction, preprocessing, and training on the training data (for example, the training images in the model training method provided in the embodiment of the present application, etc.), and finally obtains a trained neural network (such as the target model in the model training method provided in the embodiment of the present application); and the target detection method provided in the embodiment of the present application can use the above-mentioned trained neural network to input input data (for example, the target image in the target detection method provided in the embodiment of the present application, etc.) into the trained neural network to obtain output data (such as the detection result of the target image in the target detection method provided in the embodiment of the present application, etc.). It should be noted that the model training method and target detection method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0137] The target detection method provided in the embodiment of the present application can be implemented by a target model. Figure 4 is a structural diagram of the target model provided in the embodiment of the present application. As shown in Figure 4, the target model includes: a backbone network, an encoder, a decoder, and a detection network (detection head) and other modules. These multiple modules are all trained modules. Wherein, the input end of the backbone network and the first input end of the decoder are the input ends of the entire target model, the output end of the backbone network is connected to the input end of the encoder, the output end of the encoder is connected to the second input end of the decoder, the output end of the decoder is connected to the input end of the detection network, and the output end of the detection network serves as the output end of the entire target model. In order to understand the workflow of realizing target detection based on the target model, the workflow is further introduced below in conjunction with Figure 5. Figure 5 is a flow diagram of the target detection method provided in the embodiment of the present application. As shown in Figure 5, the method includes:

[0138] 501. Acquire a target image, where the target image contains an object to be detected.

[0139] In this embodiment, when target detection is required for a target image, the target image may be acquired first. The target image may contain one or more objects to be detected. For example, if target detection is required for an image, the image may be acquired first. The image may contain multiple objects to be detected, such as a cat, flowers, grass, trees, ground, and a wall.

[0140] 502. Perform feature extraction on the target image to obtain a first feature of the target image.

[0141] After obtaining the target image, the target image may be input into the target model so as to perform feature extraction on the target image through the target model, thereby obtaining a first feature of the target image.

[0142] Specifically, the target model can obtain the first feature of the target image in the following way:

[0143] After obtaining the target image, the target image can be input into the backbone network of the target model. After receiving the target image, the backbone network can perform a series of feature extraction on the target image to obtain the first feature of the target image, and send the first feature of the target image to the encoder.

[0144] For example, as shown in Figure 6 (Figure 6 is a schematic diagram of the structure of the target model provided in an embodiment of the present application), assume that the target image contains a cat, an object to be detected. The target image is input into the backbone network of the target model. The backbone network can extract features of the target image to obtain the visual features of the target image (i.e., the first features mentioned above), and send the visual features of the target image to the encoder.

[0145] 503. Encode the first feature to obtain a second feature of the target image, where the encoding includes at least one convolution and does not include processing based on an attention mechanism.

[0146] After obtaining the first feature of the target image, the target model may encode the first feature of the target image to obtain the second feature of the target image. It is worth noting that the encoding performed by the target model on the first feature of the target image may include at least one convolution, but does not include any attention-based processing (e.g., self-attention-based processing, cross-attention-based processing, etc.).

[0147] Specifically, the target model can obtain the second feature of the target image in the following way:

[0148] After obtaining the first feature of the target image, the encoder may encode the first feature of the target image to obtain the second feature of the target image, and send the second feature of the target image to the decoder. Since the encoder includes at least one convolutional layer, for example, these convolutional layers can be one or more of channel-by-channel convolution layers and point-by-point convolution layers. Therefore, the encoder may perform at least one channel-by-channel convolution and / or at least one point-by-point convolution on the first feature of the target image to obtain the second feature of the target image.

[0149] Still as in the above example, after obtaining the visual features of the target image, the encoder composed of the convolutional layer can convolve the visual features of the target image, for example, channel-by-channel convolution and / or point-by-point convolution, so as to complete feature enhancement, obtain the coding features of the target image (i.e., the aforementioned second features), and send the coding features of the target image to the decoder, wherein the coding features of the target image can also be understood as the enhanced visual features of the target image, which is conducive to the subsequent more efficient completion of target detection. It should be noted that the size of the coding features of the target image can be set here to d×H×W, that is, the number of channels of the coding features is d, the height of the coding features is H, and the width of the coding features is W. The size of the coding features of the target image can be the same as or different from the size of the target image, and there is no restriction here.

[0150] More specifically, the channel-by-channel convolutional layer contained in the encoder can be represented as follows:

[0151] The channel-by-channel convolution layer used to constitute the encoder can be one or more of a channel-by-channel standard convolution layer, a channel-by-channel deformable convolution layer, a channel-by-channel dynamic convolution layer, and the like. Accordingly, the encoding of the first feature of the target image by the encoder can include one or more of a channel-by-channel standard convolution, a channel-by-channel deformable convolution, a channel-by-channel dynamic convolution, and the like.

[0152] More specifically, the point-by-point convolutional layers contained in the encoder can be represented as follows:

[0153] The point-by-point convolution layer used to constitute the encoder can be one or more of a point-by-point standard convolution layer, a point-by-point deformable convolution layer, a point-by-point dynamic convolution layer, and the like. Accordingly, the encoding of the first feature of the target image performed by the encoder can include one or more of a point-by-point standard convolution, a point-by-point deformable convolution, a point-by-point dynamic convolution, and the like.

[0154] 504. Decode the second feature based on a preset query vector to obtain a third feature of the target image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism.

[0155] When acquiring a target image, a preset query vector (query) is also acquired and input into the target model. The query vector contains multiple randomly generated parameters (or multiple parameters with fixed values). The number of parameters (which can be set based on actual needs and are not limited here) is associated with the number of objects recognized by the target model. In other words, the number of parameters determines the number of objects recognized by the target model.

[0156] After obtaining the second feature of the target image, the target model can use the preset query vector to decode the second feature of the target image to obtain the third feature of the target image. It is worth noting that the decoding of the second feature of the target image performed by the target model may include at least one convolution, but also does not include any processing based on the attention mechanism.

[0157] Specifically, the target model can obtain the third feature of the target image in the following way:

[0158] (1) After obtaining the preset query vector and the second feature of the target image, since the decoder includes three modules, namely the self-interaction module (SIM), the cross-interaction module (CIM), and the post-processing module, the self-interaction module can first perform a first processing on the preset query vector to obtain the fourth feature of the query vector, and then send the fourth feature of the query vector to the cross-interaction module. Among them, the first processing performed by the self-interaction module on the query vector can at least include channel-by-channel convolution and point-by-point convolution, but does not include any processing based on the attention mechanism.

[0159] (2) After obtaining the fourth feature of the query vector, the cross-interaction module may perform a second processing on the second feature of the target image and the fourth feature of the query vector, thereby obtaining a fifth feature of the target image, and sending the fifth feature of the target image to the post-processing module. The second processing performed by the cross-interaction module on the second feature and the fourth feature may at least include channel-by-channel convolution, but does not include any processing based on the attention mechanism.

[0160] (3) After obtaining the fifth feature of the target image, the post-processing module can perform a third processing on the fifth feature of the target image to obtain the third feature of the target image, and send the third feature of the target image to the detection network.

[0161] More specifically, the self-interaction module can obtain the fourth feature of the query vector in the following way:

[0162] (1.1) After obtaining the preset query vector, since the self-interaction module includes at least one convolutional layer and an additive layer (skip connection layer), for example, these convolutional layers can be channel-by-channel convolution layers and point-by-point convolution layers. Therefore, the self-interaction module can first perform at least one channel-by-channel convolution and at least one point-by-point convolution on the preset query vector to obtain the sixth feature of the query vector.

[0163] (1.2) After obtaining the sixth feature of the query vector, the self-interaction module may further add the sixth feature of the query vector and the query vector to obtain the fourth feature of the query vector, and send the fourth feature of the query vector to the cross-interaction module.

[0164] For example, as shown in Figure 7 (Figure 7 is a structural diagram of the decoder provided in an embodiment of the present application, and Figure 7 is drawn based on Figure 6), the preset query vector is assumed to be a set of randomly initialized parameters. The size of the query vector is d×N×M, where N×M determines the number of detection boxes output by the detection network (a detection box represents the location information of an object, so the number of detection boxes is also the number of objects). The values ​​of N and M can be the same or different, for example, N=10, M=20, and so on.

[0165] The decoder includes a SIM module, a CIM module, and a post-processing module. The SIM module includes a channel-by-channel convolution (k×kdwconv) layer, two point-by-point convolution (1×1conv) layers, and an addition layer connected in series. After the preset query vector is input into the SIM module of the decoder, the SIM module can perform one channel-by-channel convolution and two point-by-point convolutions on the query vector in sequence to obtain the initial self-interaction feature of the query vector (i.e., the aforementioned sixth feature). Then, the SIM module can add the query vector to the initial self-interaction feature of the query vector to obtain the final self-interaction feature of the query vector (i.e., the aforementioned fourth feature), and send the final self-interaction feature of the query vector to the CIM module.

[0166] More specifically, the cross-interaction module can obtain the fifth feature of the target image in the following way:

[0167] (2.1) After obtaining the fourth feature of the query vector, the cross-interaction module can first upsample the fourth feature of the query vector to obtain the seventh feature of the query vector, because the cross-interaction module includes an upsampling layer, a fusion layer, a channel-by-channel convolution layer, and an addition layer.

[0168] (2.2) After obtaining the seventh feature of the query vector and the second feature of the target image, the cross-interaction module can fuse the second feature of the target image and the seventh feature of the query vector (for example, addition, multiplication, subtraction, and splicing, etc.) to obtain the eighth feature of the target image.

[0169] (2.3) After obtaining the eighth feature of the target image, the cross-interaction module may perform at least one channel-by-channel convolution on the eighth feature of the target image, thereby obtaining the ninth feature of the target image.

[0170] (2.4) After obtaining the ninth feature of the target image, the cross-interaction module can add the ninth feature of the target image and the seventh feature of the query vector to obtain the fifth feature of the target image, and send the fifth feature of the target image to the post-processing module. Thus, after steps (2.2) and (2.4), the cross-interaction module can fully interact with the features of the target image and the features of the query vector, which is conducive to subsequent better target detection.

[0171] Still as in the above example, the CIM module includes an upsample layer, a fusion layer, a channel-by-channel convolution (k×k dwconv) layer, and an addition layer connected in series. After obtaining the encoding features of the target image and the final self-interaction features of the query vector, the CIM module can first upsample the final self-interaction features of the query vector to obtain the upsampled self-interaction features of the query vector (i.e., the aforementioned seventh feature). The upsampling operation of the CIM module can make the size of the upsampled self-interaction features of the query vector the same as the size of the encoding features of the target image, that is, the size of the upsampled self-interaction features of the query vector is also d×H×W. Among them, the upsampling operation performed by the CIM module is shown in the following formula:

[0172] In the above formula, ο is the final self-interaction feature of the query vector, is the upsampled self-interaction feature of the query vector.

[0173] Next, the CIM module can fuse the upsampled self-interaction features of the query vector and the encoding features of the target image to obtain the fused features of the target image (i.e., the aforementioned eighth feature), and perform channel-by-channel convolution on the fused features of the target image to obtain the depth features of the target image (i.e., the aforementioned ninth feature). The depth features of the target image are then added to the upsampled self-interaction features of the query vector to obtain the cross-interaction features of the target image (i.e., the aforementioned fifth feature), and the cross-interaction features of the target image are sent to the post-processing module. The process is shown in the following formula:

[0174] In the above formula, z is the encoding feature of the target image, is the fusion feature of the target image, is the depth feature of the target image, is the cross-interaction feature of the target image.

[0175] More specifically, the post-processing module can obtain the third feature of the target image in the following manner:

[0176] (3.1) After obtaining the fifth feature of the target image, the post-processing module includes a feedforward neural network layer, an addition layer, and a pooling layer. Therefore, the post-processing module can first perform feedforward neural network-based processing (e.g., convolution and / or full connection, etc.) on the fifth feature to obtain the tenth feature of the target image.

[0177] (3.2) After obtaining the tenth feature of the target image, the post-processing module may add the fifth feature of the target image and the tenth feature of the target image to obtain the eleventh feature of the target image.

[0178] (3.3) After obtaining the eleventh feature of the target image, the post-processing module can pool the eleventh feature of the target image to obtain the third feature of the target image, and send the third feature of the target image to the detection network.

[0179] Still as in the above example, the post-processing module includes a feedforward neural network (FFN) layer, an addition layer, and a pooling layer connected in series. After obtaining the cross-interaction features of the target image, the post-processing module can process the cross-interaction features of the target image to obtain the feedforward features of the target image (i.e., the aforementioned tenth feature), and add the feedforward features of the target image to the cross-interaction features of the target image to obtain the initial decoding features of the target image (i.e., the aforementioned eleventh feature), and then pool the initial decoding features of the target image to obtain the final decoding features of the target image (i.e., the aforementioned third feature), and send the final decoding features of the target image to the detection network. The size of the feature is d×N×M. The process is shown in the following formula:

[0180] In the above formula, is the feedforward feature of the target image, is the initial decoding feature of the target image, is the final decoded feature of the target image.

[0181] 505. Based on the third feature, obtain a detection result of the target image, where the detection result is used to determine the location information and the category of the object.

[0182] After obtaining the third feature of the target image, the target model (detection network) can further process the third feature of the target image to obtain the detection result of the target image. It should be noted that the detection result includes the position information of at least one object detected by the model, the category of the at least one object and the confidence of the at least one object. Since the model will not only recognize some necessary objects (for example, foreground objects, such as cats, flowers, grass, trees, etc.), but also recognize some unnecessary objects (for example, background objects, such as the ground, walls, etc.), unnecessary objects often have lower confidence, so the position information and category of objects with lower confidence can be eliminated, and the position information and category of objects with higher confidence can be retained as the position information and category of the object finally detected by the model. At this point, the target detection for the target image is completed.

[0183] Still as in the above example, the final decoded features of the target image are obtained, and the detection network can classify and regress the final decoded features of the target image to obtain the final detection results, which include N×M detection frames, the categories of objects in the N×M detection frames, and the confidence of the N×M detection frames. Then, the detection frames with low confidence (for example, below a certain threshold, the size of the threshold can be set according to actual needs and is not limited here) can be eliminated, and the remaining detection frames with higher confidence and the categories of objects in these detection frames are retained. At this point, the target detection of the target image is successfully completed.

[0184] It should be understood that in this embodiment, the target model is only schematically introduced as including one decoder. In actual applications, the target model may also include multiple decoders connected in series, as shown in Figure 8 (Figure 8 is another structural schematic diagram of the target model provided in an embodiment of the present application). Multiple decoders are arranged between the encoder and the detection network. The input of the first decoder is the output of the encoder and a preset query vector. The input of the second decoder is the output of the encoder, the output of the first decoder (i.e., the third feature), and a preset query vector (the input of the SIM module of the second decoder is the output of the first decoder and a preset query vector), ..., and the input of the last decoder is the output of the encoder, the output of the second-to-last decoder, and a preset query vector. Then, the detection network can obtain the detection result of the target image based on the output of the last decoder.

[0185] In addition, the target model provided in the embodiment of the present application (for example, DECO in Table 1) can be compared with the models of related technologies (for example, the models other than DECO in Table 1, such as FCOS, DETR, etc.) in terms of detection index (AP) and inference speed (FPS). The comparison results are shown in Table 1:

[0186] Table 1

[0187] Based on Table 1, a curve diagram as shown in Figure 9 can be obtained (Figure 9 is a diagram of the comparison results provided by the embodiment of the present application). It can be seen that the performance of the target model provided by the embodiment of the present application is better than the performance of the model provided by the related art.

[0188] In an embodiment of the present application, when target detection is required for a target image, a target image containing the object to be detected can be first obtained and input into a target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location information and category. At this point, target detection for the target model is complete. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection. Even if the computing power of the device equipped with the target model is low, the target detection task can be completed quickly, improving the efficiency of the target detection task and thereby enhancing the user experience.

[0189] The above is a detailed description of the target detection method provided by the embodiment of the present application. The following is an introduction to the model training method provided by the embodiment of the present application. Figure 10 is a flow chart of the model training method provided by the embodiment of the present application. As shown in Figure 10, the method includes:

[0190] 1001. Obtain a training image, where the training image contains an object to be detected.

[0191] In this embodiment, when training the model to be trained is required, a batch of training data can be first obtained, which includes training images. It should be noted that the true detection results of the training images are known and include the true position information of at least one object to be detected, the true category of the at least one object, and the true confidence level of the at least one object.

[0192] 1002. Process the training image through the model to be trained to obtain a detection result of the training image, and the detection result is used to determine the location information of the object and the category of the object. The model to be trained is used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, wherein the encoding includes at least one convolution and does not include processing based on an attention mechanism; decode the second feature based on a preset query vector to obtain a third feature of the training image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism; and obtain the detection result of the training image based on the third feature.

[0193] After obtaining the training image, the training image can be input into the model to be trained so that the training image can be processed by the model to obtain a (predicted) detection result of the training image. The detection result of the training image includes the (predicted) position information of the at least one object, the (predicted) category of the at least one object, and the (predicted) confidence of the at least one object. The model to be trained is used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, wherein the encoding includes at least one convolution and does not include processing based on an attention mechanism; decode the second feature based on a preset query vector to obtain a third feature of the training image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism; and obtain a detection result of the training image based on the third feature.

[0194] In one possible implementation, the encoding includes at least one of the following: channel-by-channel convolution or point-by-point convolution.

[0195] In one possible implementation, decoding the second feature based on a preset query vector to obtain the third feature of the training image includes: performing a first processing on the preset query vector to obtain a fourth feature of the query vector, the first processing including channel-by-channel convolution and point-by-point convolution; performing a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, the second processing including channel-by-channel convolution; and performing a third processing on the fifth feature to obtain the third feature of the training image.

[0196] In one possible implementation, performing a first processing on a preset query vector to obtain a fourth feature of the query vector includes: performing channel-by-channel convolution and point-by-point convolution on the preset query vector to obtain a sixth feature of the query vector; and adding the sixth feature and the query vector to obtain a fourth feature of the query vector.

[0197] In one possible implementation, performing a second processing on the second feature and the fourth feature to obtain the fifth feature of the training image includes: upsampling the fourth feature to obtain the seventh feature of the query vector; fusing the second feature and the seventh feature to obtain the eighth feature of the training image; performing channel-by-channel convolution on the eighth feature to obtain the ninth feature of the training image; and adding the ninth feature and the seventh feature to obtain the fifth feature of the training image.

[0198] In one possible implementation, performing a third processing on the fifth feature to obtain the third feature of the training image includes: performing a feedforward neural network-based processing on the fifth feature to obtain the tenth feature of the training image; adding the fifth feature and the tenth feature to obtain the eleventh feature of the training image; and pooling the eleventh feature to obtain the third feature of the training image.

[0199] In one possible implementation, the channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

[0200] In one possible implementation, the point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

[0201] In one possible implementation, the query vector includes multiple parameters, and the number of parameters is related to the number of objects.

[0202] It should be understood that the introduction to step 1002 can refer to the relevant description of steps 502 to 505 in the embodiment shown in Figure 5, and will not be repeated here.

[0203] 1003. Based on the detection results and the actual detection results of the training images, the to-be-trained model is trained to obtain a target model.

[0204] After obtaining the detection results of the training image, since the true detection results of the training image are known, the detection results of the training image and the true detection results of the training image can be calculated through a preset loss function to obtain the target loss. The target loss is used to indicate the difference between the detection results of the training image and the true detection results of the training image.

[0205] After obtaining the target loss, the target loss can be used to update the parameters of the model to be trained, thereby obtaining the model to be trained after the updated parameters, and the next batch of training data can be used to continue training the model to be trained after the updated parameters until the model training conditions are met (for example, the target loss converges, etc.), and the target model in the embodiment shown in Figure 5 is obtained.

[0206] The target model trained in the embodiments of the present application has target detection capabilities. Specifically, when target detection is required for a target image, a target image containing the object to be detected can be first obtained and input into the target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location information and category. At this point, target detection for the target model is complete. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection. Even if the computing power of the device equipped with the target model is low, the target detection task can be completed quickly, improving the efficiency of the target detection task and, in turn, enhancing the user experience.

[0207] The above is a detailed description of the target detection method and model training method provided by the embodiment of the present application. The target detection device and model training device provided by the embodiment of the present application will be introduced below. Figure 11 is a structural schematic diagram of the target detection device provided by the embodiment of the present application. As shown in Figure 11, the device includes a target model. The device includes:

[0208] An acquisition module 1101 is used to acquire a target image, where the target image contains an object to be detected;

[0209] An extraction module 1102 is configured to extract features from a target image to obtain a first feature of the target image;

[0210] An encoding module 1103 is configured to encode the first feature to obtain a second feature of the target image, wherein the encoding includes at least one convolution and does not include processing based on an attention mechanism;

[0211] A decoding module 1104 is configured to decode the second feature based on a preset query vector to obtain a third feature of the target image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism;

[0212] The detection module 1105 is configured to obtain a detection result of the target image based on the third feature, and the detection result is used to determine the location information and the category of the object.

[0213] In an embodiment of the present application, when target detection is required for a target image, a target image containing the object to be detected can be first obtained and input into a target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location information and category. At this point, target detection for the target model is complete. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection. Even if the computing power of the device equipped with the target model is low, the target detection task can be completed quickly, improving the efficiency of the target detection task and thereby enhancing the user experience.

[0214] In one possible implementation, the encoding includes at least one of the following: channel-by-channel convolution or point-by-point convolution.

[0215] In one possible implementation, the decoding module 1104 is used to: perform a first processing on a preset query vector to obtain a fourth feature of the query vector, where the first processing includes channel-by-channel convolution and point-by-point convolution; perform a second processing on the second feature and the fourth feature to obtain a fifth feature of the target image, where the second processing includes channel-by-channel convolution; and perform a third processing on the fifth feature to obtain a third feature of the target image.

[0216] In one possible implementation, the decoding module 1104 is configured to: perform channel-by-channel convolution and point-by-point convolution on a preset query vector to obtain a sixth feature of the query vector; and add the sixth feature and the query vector to obtain a fourth feature of the query vector.

[0217] In one possible implementation, the decoding module 1104 is used to: upsample the fourth feature to obtain the seventh feature of the query vector; fuse the second feature and the seventh feature to obtain the eighth feature of the target image; perform channel-by-channel convolution on the eighth feature to obtain the ninth feature of the target image; and add the ninth feature and the seventh feature to obtain the fifth feature of the target image.

[0218] In one possible implementation, the decoding module 1104 is used to: process the fifth feature based on a feedforward neural network to obtain a tenth feature of the target image; add the fifth feature and the tenth feature to obtain an eleventh feature of the target image; and pool the eleventh feature to obtain a third feature of the target image.

[0219] In one possible implementation, the channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

[0220] In one possible implementation, the point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

[0221] In one possible implementation, the query vector includes multiple parameters, and the number of parameters is related to the number of objects.

[0222] FIG12 is a schematic diagram of a structure of a model training device provided in an embodiment of the present application. As shown in FIG12 , the device includes:

[0223] An acquisition module 1201 is configured to acquire a training image, where the training image contains an object to be detected;

[0224] The processing module 1202 is configured to process the training image using the model to be trained to obtain a detection result of the training image, where the detection result is used to determine the location information and category of the object. The model to be trained is configured to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, where the encoding includes at least one convolution and does not include processing based on an attention mechanism; decode the second feature based on a preset query vector to obtain a third feature of the training image, where the decoding includes at least one convolution and does not include processing based on an attention mechanism; and obtain a detection result of the training image based on the third feature.

[0225] The training module 1203 is used to train the to-be-trained model based on the detection results and the actual detection results of the training images to obtain a target model.

[0226] The target model trained in the embodiments of the present application has target detection capabilities. Specifically, when target detection is required for a target image, a target image containing the object to be detected can be first obtained and input into the target model. The target model then performs feature extraction on the target image to obtain a first feature of the target image. The target model then encodes the first feature of the target image to obtain a second feature of the target image. Subsequently, the target model decodes the second feature of the target image based on a preset query vector to obtain a third feature of the target image. Finally, based on the third feature, the target model obtains a detection result for the target image. The detection result can be used to determine the object's location information and category. At this point, target detection for the target model is complete. In the aforementioned process, the primary operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection. Even if the computing power of the device equipped with the target model is low, the target detection task can be completed quickly, improving the efficiency of the target detection task and, in turn, enhancing the user experience.

[0227] In one possible implementation, the encoding includes at least one of the following: channel-by-channel convolution or point-by-point convolution.

[0228] In one possible implementation, the model to be trained is used to: perform a first processing on a preset query vector to obtain a fourth feature of the query vector, where the first processing includes channel-by-channel convolution and point-by-point convolution; perform a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, where the second processing includes channel-by-channel convolution; and perform a third processing on the fifth feature to obtain a third feature of the training image.

[0229] In one possible implementation, the model to be trained is used to: perform channel-by-channel convolution and point-by-point convolution on a preset query vector to obtain a sixth feature of the query vector; and add the sixth feature and the query vector to obtain a fourth feature of the query vector.

[0230] In one possible implementation, the model to be trained is used to: upsample the fourth feature to obtain the seventh feature of the query vector; fuse the second feature and the seventh feature to obtain the eighth feature of the training image; perform channel-by-channel convolution on the eighth feature to obtain the ninth feature of the training image; and add the ninth feature and the seventh feature to obtain the fifth feature of the training image.

[0231] In one possible implementation, the model to be trained is used to: process the fifth feature based on a feedforward neural network to obtain the tenth feature of the training image; add the fifth feature and the tenth feature to obtain the eleventh feature of the training image; and pool the eleventh feature to obtain the third feature of the training image.

[0232] In one possible implementation, the channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

[0233] In one possible implementation, the point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

[0234] In one possible implementation, the query vector includes multiple parameters, and the number of parameters is related to the number of objects.

[0235] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.

[0236] The embodiment of the present application also relates to an execution device, and Figure 13 is a structural diagram of the execution device provided by the embodiment of the present application. As shown in Figure 13, the execution device 1300 can be specifically manifested as a mobile phone, a tablet, a laptop computer, an intelligent wearable device, a server, etc., which are not limited here. Among them, the target detection device described in the embodiment corresponding to Figure 11 can be deployed on the execution device 1300 to implement the function of target detection in the embodiment corresponding to Figure 5. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (wherein the number of processors 1303 in the execution device 1300 can be one or more, and Figure 13 takes one processor as an example), wherein the processor 1303 may include an application processor 13031 and a communication processor 13032. In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303 and the memory 1304 can be connected via a bus or other means.

[0237] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0238] Processor 1303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0239] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1303. Processor 1303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1303. The above processor 1303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1303 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1304, and processor 1303 reads the information in memory 1304 and, in conjunction with its hardware, completes the steps of the above method.

[0240] Receiver 1301 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1302 can be used to output digital or character information through the first interface. Transmitter 1302 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1302 can also include a display device such as a display screen.

[0241] In an embodiment of the present application, in one case, the processor 1303 is configured to complete target detection for the target image using the target model in the embodiment corresponding to FIG. 5 .

[0242] The present application also relates to a training device. FIG14 is a schematic diagram of the structure of the training device provided by the present application. As shown in FIG14 , the training device 1400 is implemented by one or more servers. The training device 1400 may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1414 (e.g., one or more processors) and memory 1432, and one or more storage media 1430 (e.g., one or more mass storage devices) storing application programs 1442 or data 1444. The memory 1432 and storage medium 1430 may be temporary storage or permanent storage. The program stored in the storage medium 1430 may include one or more modules (not shown), each module may include a series of instruction operations in the training device. Furthermore, the CPU 1414 may be configured to communicate with the storage medium 1430 to execute the series of instruction operations in the storage medium 1430 on the training device 1400.

[0243] The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458; or, one or more operating systems 1441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0244] Specifically, the training device can execute the model training method in the embodiment corresponding to Figure 10 to obtain the target model.

[0245] An embodiment of the present application also relates to a computer storage medium, which stores a program for signal processing. When the computer storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0246] An embodiment of the present application also relates to a computer program product, which stores instructions that, when executed by a computer, enable the computer to execute the steps executed by the aforementioned execution device, or enable the computer to execute the steps executed by the aforementioned training device.

[0247] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0248] Specifically, see Figure 15 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip may be a neural network processor (NPU) 1500. NPU 1500 is mounted on a host CPU (host CPU) as a coprocessor, with tasks assigned by the host CPU. The core of the NPU is arithmetic circuit 1503, which is controlled by controller 1504 to extract matrix data from memory and perform multiplication operations.

[0249] In some implementations, arithmetic circuit 1503 includes multiple processing units (PEs). In some implementations, arithmetic circuit 1503 is a two-dimensional systolic array. Arithmetic circuit 1503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1503 is a general-purpose matrix processor.

[0250] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1502 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1501 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1508.

[0251] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 through the Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 through the DMAC.

[0252] BIU stands for Bus Interface Unit, i.e., bus interface unit 1513 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1509 .

[0253] The bus interface unit 1513 (BIU) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0254] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or transfer weight data to the weight memory 1502 or transfer input data to the input memory 1501.

[0255] The vector calculation unit 1507 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of the predicted label plane.

[0256] In some implementations, the vector calculation unit 1507 can store the processed output vector to the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation of the predicted label plane extracted by the convolution layer, or another example is a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1507 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in subsequent layers in a neural network.

[0257] An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;

[0258] Unified memory 1506, input memory 1501, weight memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0259] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0260] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0261] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0262] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0263] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A target detection method, characterized in that: The method is implemented by a target model, and the method comprises: Acquire a target image, wherein the target image includes an object to be detected; Performing feature extraction on the target image to obtain a first feature of the target image; Encoding the first feature to obtain a second feature of the target image, wherein the encoding includes at least one convolution and does not include processing based on an attention mechanism; Decoding the second feature based on a preset query vector to obtain a third feature of the target image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism; Based on the third feature, a detection result of the target image is obtained, and the detection result is used to determine the position information of the object and the category of the object.

2. The method according to claim 1, characterized in that The encoding includes at least one of the following: channel-by-channel convolution or point-by-point convolution.

3. The method according to claim 1 or 2, characterized in that: The decoding of the second feature based on the preset query vector to obtain the third feature of the target image includes: Performing a first processing on a preset query vector to obtain a fourth feature of the query vector, wherein the first processing includes channel-by-channel convolution and point-by-point convolution; Performing a second processing on the second feature and the fourth feature to obtain a fifth feature of the target image, wherein the second processing includes channel-by-channel convolution; Perform a third processing on the fifth feature to obtain a third feature of the target image.

4. The method according to claim 3, characterized in that: The performing the first processing on the preset query vector to obtain the fourth feature of the query vector includes: Performing channel-by-channel convolution and point-by-point convolution on a preset query vector to obtain a sixth feature of the query vector; The sixth feature and the query vector are added to obtain a fourth feature of the query vector.

5. The method according to claim 3 or 4, characterized in that: The performing a second processing on the second feature and the fourth feature to obtain a fifth feature of the target image includes: Upsampling the fourth feature to obtain a seventh feature of the query vector; fusing the second feature and the seventh feature to obtain an eighth feature of the target image; Performing channel-by-channel convolution on the eighth feature to obtain a ninth feature of the target image; The ninth feature and the seventh feature are added to obtain a fifth feature of the target image.

6. The method according to any one of claims 3 to 5, characterized in that: The performing a third processing on the fifth feature to obtain the third feature of the target image includes: Processing the fifth feature based on a feedforward neural network to obtain a tenth feature of the target image; Adding the fifth feature and the tenth feature to obtain an eleventh feature of the target image; Pooling is performed on the eleventh feature to obtain a third feature of the target image.

7. The method according to any one of claims 2 to 6, characterized in that: The channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

8. The method according to any one of claims 2 to 7, characterized in that: The point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

9. The method according to any one of claims 1 to 8, characterized in that: The query vector includes a plurality of parameters, and the number of the parameters is associated with the number of the objects.

10. A model training method, characterized in that: The method comprises: Acquire a training image, wherein the training image contains an object to be detected; The training image is processed by the model to be trained to obtain a detection result of the training image, and the detection result is used to determine the location information of the object and the category of the object. The model to be trained is used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, wherein the encoding includes at least one convolution and does not include processing based on an attention mechanism; decode the second feature based on a preset query vector to obtain the second feature of the training image; a third feature of the training image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism; and based on the third feature, obtaining a detection result of the training image; Based on the detection result and the real detection result of the training image, the model to be trained is trained to obtain a target model.

11. The method according to claim 10, characterized in that The encoding includes at least one of the following: channel-by-channel convolution or point-by-point convolution.

12. The method according to claim 10 or 11, characterized in that: The decoding of the second feature based on the preset query vector to obtain the third feature of the training image includes: Performing a first processing on a preset query vector to obtain a fourth feature of the query vector, wherein the first processing includes channel-by-channel convolution and point-by-point convolution; Performing a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, wherein the second processing includes channel-by-channel convolution; Perform a third processing on the fifth feature to obtain a third feature of the training image.

13. The method according to claim 12, characterized in that The performing the first processing on the preset query vector to obtain the fourth feature of the query vector includes: Performing channel-by-channel convolution and point-by-point convolution on a preset query vector to obtain a sixth feature of the query vector; The sixth feature and the query vector are added to obtain a fourth feature of the query vector.

14. The method according to claim 12 or 13, characterized in that The performing a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image includes: Upsampling the fourth feature to obtain a seventh feature of the query vector; fusing the second feature and the seventh feature to obtain an eighth feature of the training image; Performing channel-by-channel convolution on the eighth feature to obtain a ninth feature of the training image; The ninth feature and the seventh feature are added to obtain a fifth feature of the training image.

15. The method according to any one of claims 12 to 14, characterized in that: The performing a third processing on the fifth feature to obtain the third feature of the training image includes: Processing the fifth feature based on a feedforward neural network to obtain a tenth feature of the training image; Adding the fifth feature and the tenth feature to obtain an eleventh feature of the training image; Pooling is performed on the eleventh feature to obtain a third feature of the training image.

16. The method according to any one of claims 11 to 15, characterized in that: The channel-by-channel convolution includes at least one of the following: channel-by-channel standard convolution, channel-by-channel deformable convolution, or channel-by-channel dynamic convolution.

17. The method according to any one of claims 11 to 16, characterized in that: The point-by-point convolution includes at least one of the following: point-by-point standard convolution, point-by-point deformable convolution, or point-by-point dynamic convolution.

18. The method according to any one of claims 10 to 17, characterized in that: The query vector includes a plurality of parameters, and the number of the parameters is associated with the number of the objects.

19. A target detection device, characterized in that: The device comprises a target model, and the device comprises: An acquisition module, used for acquiring a target image, wherein the target image includes an object to be detected; An extraction module, used for performing feature extraction on the target image to obtain a first feature of the target image; an encoding module, configured to encode the first feature to obtain a second feature of the target image, wherein the encoding includes at least one convolution and does not include processing based on an attention mechanism; A decoding module, configured to decode the second feature based on a preset query vector to obtain a third feature of the target image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism; A detection module is used to obtain a detection result of the target image based on the third feature, wherein the detection result is used to determine the location information of the object and the category of the object.

20. A model training device, characterized in that: The device comprises: An acquisition module, used for acquiring a training image, wherein the training image contains an object to be detected; A processing module is used to process the training image through the model to be trained to obtain the detection result of the training image, and the detection result is used to determine the location information of the object and the category of the object. The model to be trained is used to: Feature extraction to obtain a first feature of the training image; encoding the first feature to obtain a second feature of the training image, wherein the encoding includes at least one convolution and does not include processing based on an attention mechanism; decoding the second feature based on a preset query vector to obtain a third feature of the training image, wherein the decoding includes at least one convolution and does not include processing based on an attention mechanism; and obtaining a detection result of the training image based on the third feature; The training module is used to train the model to be trained based on the detection result and the real detection result of the training image to obtain a target model.

21. A target detection device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the target detection device performs the method according to any one of claims 1 to 18.

22. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 18.

23. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 18.

Images