A target detection method and related device

Abstract

The application discloses a target detection method and a related device, which can quickly complete a target detection task, improve the completion efficiency of the target detection task, and further improve user experience. The method of the application comprises the following steps: when target detection needs to be performed on a target image, the target image containing an object to be detected is acquired first, and then the target image is input into a target model. Next, the target model performs feature extraction on the target image, thereby obtaining first features of the target image. Then, the target model encodes the first features of the target image, thereby obtaining second features of the target image. Subsequently, the target model decodes the second features of the target image based on a preset query vector, thereby obtaining third features of the target image. Finally, the target model acquires a detection result of the target image based on the third features, and the detection result can be used to determine position information of the object and a category of the object. Thus, the target detection on the target model is completed.

Description

Technical Field

[0001] This application relates to artificial intelligence (AI), and more particularly to a target detection method and related equipment. Background Technology

[0002] Object detection is one of the most fundamental computer vision tasks and is crucial for many practical applications. In recent years, the transformer model and its variants have demonstrated outstanding performance in image classification tasks, so they can be transferred to object detection tasks to improve the processing results.

[0003] Currently, when detecting objects within a target image, the target image is first input into a transformer model. The transformer model then extracts initial features from the target image. Next, it processes these initial features using a self-attention mechanism to obtain intermediate features. Then, it processes these intermediate features using both self-attention and cross-attention mechanisms to obtain the final features. Finally, the transformer model uses these final features to obtain the detection result, thus determining the object's location and category.

[0004] In the process described above, since the transformer model is mainly built on the attention mechanism, it requires a lot of computational cost to perform object detection. If the computing power of the device equipped with the transformer model is low, the object detection speed will be slow, affecting the efficiency of object detection and resulting in a poor user experience. Summary of the Invention

[0005] This application provides a target detection method and related equipment, which can quickly complete target detection tasks, improve the efficiency of target detection tasks, and thus improve user experience.

[0006] A first aspect of this application provides a target detection method, which can be implemented through a target model, and the method includes:

[0007] When object detection is required on a target image, the target image and a pre-defined query vector can be obtained first. The target image contains one or more objects to be detected.

[0008] After obtaining the target image and query vector, these can be input into the target model. The target model first extracts features from the target image to obtain its first feature. Then, it encodes this first feature to obtain its second feature. Next, using the pre-defined query vector, it decodes the second feature to obtain its third feature. Finally, the target model further processes this third feature to obtain the detection result for the target image.

[0009] It is worth noting that the encoding of the first feature of the target image performed by the target model may include at least one convolution, but does not include any attention-based processing. Furthermore, the decoding of the second feature of the target image performed by the target model may include at least one convolution, but also does not include any attention-based processing.

[0010] The detection results of the target image include the location information of at least one object detected by the model, the category of at least one object, and the confidence level of at least one object. Therefore, the location information and category of objects with low confidence can be removed, while the location information and category of objects with high confidence can be retained as the final location information and category of the objects detected by the model. This completes the target detection for the target image.

[0011] As can be seen from the above method, when object detection is required on a target image, the target image containing the object to be detected can be acquired first and input into the target model. Next, the target model can extract features from the target image to obtain the first feature. Then, the target model can encode the first feature to obtain the second feature. Subsequently, the target model can decode the second feature based on a pre-set query vector to obtain the third feature. Finally, the target model can obtain the detection result based on the third feature, which can be used to determine the object's location and category. This completes the object detection for the target model. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations involve at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during object detection, enabling it to quickly complete the object detection task even on devices with lower computing power, improving the efficiency of object detection and thus enhancing the user experience.

[0012] In one possible implementation, the encoding includes at least one of the following: channel-wise convolution or pointwise convolution. In the aforementioned implementation, the target model can perform at least one channel-wise convolution and / or at least one pointwise convolution on the first feature of the target image to obtain the second feature of the target image.

[0013] In one possible implementation, decoding the second feature based on a preset query vector to obtain the third feature of the target image includes: performing a first processing on the preset query vector to obtain a fourth feature of the query vector, the first processing including channel-wise convolution and pointwise convolution; performing a second processing on the second and fourth features to obtain a fifth feature of the target image, the second processing including channel-wise convolution; and performing a third processing on the fifth feature to obtain the third feature of the target image. In the aforementioned implementation, after obtaining the preset query vector and the second feature of the target image, the target model can first perform the first processing on the preset query vector to obtain the fourth feature of the query vector. The first processing performed by the target model on the query vector may include at least channel-wise convolution and pointwise convolution, but does not include any attention-based processing. After obtaining the fourth feature of the query vector, the target model can perform a second processing on the second feature of the target image and the fourth feature of the query vector to obtain the fifth feature of the target image. The second processing performed by the target model on the second and fourth features may include at least channel-wise convolution, but does not include any attention-based processing. After obtaining the fifth feature of the target image, the target model can perform a third processing on the fifth feature of the target image to obtain the third feature of the target image.

[0014] In one possible implementation, the fourth feature of the query vector is obtained by performing a first processing step on the preset query vector, including: performing channel-wise convolution and pointwise convolution on the preset query vector to obtain a sixth feature; and adding the sixth feature to the query vector to obtain the fourth feature. In the aforementioned implementation, after obtaining the preset query vector, the target model can first perform at least one channel-wise convolution and at least one pointwise convolution on the preset query vector to obtain the sixth feature. After obtaining the sixth feature, the target model can also add the sixth feature to the query vector to obtain the fourth feature.

[0015] In one possible implementation, the second processing of the second and fourth features to obtain the fifth feature of the target image includes: upsampling the fourth feature to obtain the seventh feature of the query vector; fusing the second and seventh features to obtain the eighth feature of the target image; performing channel-wise convolution on the eighth feature to obtain the ninth feature of the target image; and adding the ninth feature and the seventh feature to obtain the fifth feature of the target image. In the aforementioned implementation, after obtaining the fourth feature of the query vector, the target model can first upsample the fourth feature of the query vector to obtain the seventh feature of the query vector. After obtaining the seventh feature of the query vector and the second feature of the target image, the target model can fuse the second feature of the target image and the seventh feature of the query vector to obtain the eighth feature of the target image. After obtaining the eighth feature of the target image, the target model can perform at least one channel-wise convolution on the eighth feature of the target image to obtain the ninth feature of the target image. After obtaining the ninth feature of the target image, the target model can add the ninth feature of the target image and the seventh feature of the query vector to obtain the fifth feature of the target image.

[0016] In one possible implementation, the third processing of the fifth feature to obtain the third feature of the target image includes: processing the fifth feature based on a feedforward neural network to obtain the tenth feature of the target image; adding the fifth and tenth features to obtain the eleventh feature of the target image; and pooling the eleventh feature to obtain the third feature of the target image. In the aforementioned implementation, after obtaining the fifth feature of the target image, the target model can first process the fifth feature based on a feedforward neural network to obtain the tenth feature of the target image. After obtaining the tenth feature of the target image, the target model can add the fifth and tenth features of the target image to obtain the eleventh feature of the target image. After obtaining the eleventh feature of the target image, the target model can pool the eleventh feature of the target image to obtain the third feature of the target image.

[0017] In one possible implementation, channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

[0018] In one possible implementation, pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

[0019] In one possible implementation, the query vector contains multiple parameters, the number of which is related to the number of objects.

[0020] A second aspect of this application provides a model training method, comprising: acquiring a training image containing an object to be detected; processing the training image using a model to be trained to obtain a detection result of the training image, the detection result being used to determine the location information and category of the object; the model to be trained being used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, the encoding including at least one convolution and not including attention-based processing; decode the second feature based on a preset query vector to obtain a third feature of the training image, the decoding including at least one convolution and not including attention-based processing; acquiring the detection result of the training image based on the third feature; and training the model to be trained based on the detection result and the true detection result of the training image to obtain a target model.

[0021] The target model trained using the above method possesses object detection capabilities. Specifically, when object detection is required on a target image, a target image containing the object to be detected is first acquired and input into the target model. Next, the target model extracts features from the target image to obtain its first feature. Then, the target model encodes the first feature to obtain its second feature. Subsequently, the target model decodes the second feature based on a pre-defined query vector to obtain its third feature. Finally, the target model uses the third feature to obtain the detection result of the target image, which can be used to determine the object's location and category. This completes the object detection process for the target model. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations involve at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during object detection, enabling it to quickly complete the object detection task even on devices with lower computing power, thus improving the efficiency of object detection and ultimately enhancing the user experience.

[0022] In one possible implementation, the encoding includes at least one of the following: channel-wise convolution or point-wise convolution.

[0023] In one possible implementation, decoding the second feature based on a preset query vector to obtain the third feature of the training image includes: performing a first processing on the preset query vector to obtain a fourth feature of the query vector, the first processing including channel-wise convolution and point-wise convolution; performing a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, the second processing including channel-wise convolution; and performing a third processing on the fifth feature to obtain the third feature of the training image.

[0024] In one possible implementation, the first processing of the preset query vector to obtain the fourth feature of the query vector includes: performing channel-wise convolution and point-wise convolution on the preset query vector to obtain the sixth feature of the query vector; and adding the sixth feature and the query vector to obtain the fourth feature of the query vector.

[0025] In one possible implementation, the second processing of the second and fourth features to obtain the fifth feature of the training image includes: upsampling the fourth feature to obtain the seventh feature of the query vector; fusing the second and seventh features to obtain the eighth feature of the training image; performing channel-wise convolution on the eighth feature to obtain the ninth feature of the training image; and adding the ninth and seventh features to obtain the fifth feature of the training image.

[0026] In one possible implementation, the third processing of the fifth feature to obtain the third feature of the training image includes: processing the fifth feature based on a feedforward neural network to obtain the tenth feature of the training image; adding the fifth feature and the tenth feature to obtain the eleventh feature of the training image; and pooling the eleventh feature to obtain the third feature of the training image.

[0027] In one possible implementation, channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

[0028] In one possible implementation, pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

[0029] In one possible implementation, the query vector contains multiple parameters, the number of which is related to the number of objects.

[0030] A third aspect of this application provides a target detection apparatus comprising a target model. The apparatus includes: an acquisition module for acquiring a target image containing an object to be detected; an extraction module for extracting features from the target image to obtain a first feature of the target image; an encoding module for encoding the first feature to obtain a second feature of the target image, wherein the encoding includes at least one convolution and does not include attention-based processing; a decoding module for decoding the second feature based on a preset query vector to obtain a third feature of the target image, wherein the decoding includes at least one convolution and does not include attention-based processing; and a detection module for acquiring a detection result of the target image based on the third feature, wherein the detection result is used to determine the object's location information and object category.

[0031] As can be seen from the above device, when object detection is required on a target image, a target image containing the object to be detected can be acquired first and input into the target model. Next, the target model can extract features from the target image to obtain the first feature. Then, the target model can encode the first feature to obtain the second feature. Subsequently, the target model can decode the second feature based on a preset query vector to obtain the third feature. Finally, the target model can obtain the detection result of the target image based on the third feature. The detection result can be used to determine the object's location information and category. Thus, object detection for the target model is completed. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations involve at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during object detection. Even if the device equipped with the target model has low computing power, it can quickly complete the object detection task, improving the efficiency of the object detection task and thus enhancing the user experience.

[0032] In one possible implementation, the encoding includes at least one of the following: channel-wise convolution or point-wise convolution.

[0033] In one possible implementation, the decoding module is configured to: perform a first processing on a preset query vector to obtain a fourth feature of the query vector, the first processing including channel-wise convolution and point-wise convolution; perform a second processing on the second feature and the fourth feature to obtain a fifth feature of the target image, the second processing including channel-wise convolution; and perform a third processing on the fifth feature to obtain a third feature of the target image.

[0034] In one possible implementation, the decoding module is used to: perform channel-wise convolution and point-wise convolution on a preset query vector to obtain a sixth feature of the query vector; and add the sixth feature and the query vector to obtain a fourth feature of the query vector.

[0035] In one possible implementation, the decoding module is used to: upsample the fourth feature to obtain the seventh feature of the query vector; fuse the second feature and the seventh feature to obtain the eighth feature of the target image; perform channel-wise convolution on the eighth feature to obtain the ninth feature of the target image; and add the ninth feature and the seventh feature to obtain the fifth feature of the target image.

[0036] In one possible implementation, the decoding module is used to: process the fifth feature based on a feedforward neural network to obtain the tenth feature of the target image; add the fifth feature and the tenth feature to obtain the eleventh feature of the target image; and pool the eleventh feature to obtain the third feature of the target image.

[0037] In one possible implementation, channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

[0038] In one possible implementation, pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

[0039] In one possible implementation, the query vector contains multiple parameters, the number of which is related to the number of objects.

[0040] A fourth aspect of this application provides a model training apparatus, comprising: an acquisition module for acquiring a training image containing an object to be detected; a processing module for processing the training image using a model to be trained to obtain a detection result of the training image, the detection result being used to determine the location information and category of the object; the model to be trained being configured to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, the encoding including at least one convolution and not including attention-based processing; decode the second feature based on a preset query vector to obtain a third feature of the training image, the decoding including at least one convolution and not including attention-based processing; and acquire the detection result of the training image based on the third feature; and a training module for training the model to be trained based on the detection result and the actual detection result of the training image to obtain a target model.

[0041] The target model trained in this embodiment has the function of object detection. Specifically, when object detection is required on a target image, a target image containing the object to be detected can be obtained first, and the target image can be input into the target model. Then, the target model can extract features from the target image to obtain the first feature of the target image. Then, the target model can encode the first feature of the target image to obtain the second feature of the target image. Subsequently, the target model can decode the second feature of the target image based on a preset query vector to obtain the third feature of the target image. Finally, the target model can obtain the detection result of the target image based on the third feature. The detection result can be used to determine the location information and category of the object. Thus, the object detection for the target model is completed. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost in the process of performing object detection. Even if the computing power of the device equipped with the target model is low, the object detection task can be completed quickly, improving the efficiency of the object detection task and thus improving the user experience.

[0042] In one possible implementation, the encoding includes at least one of the following: channel-wise convolution or point-wise convolution.

[0043] In one possible implementation, the model to be trained is used to: perform a first processing on a preset query vector to obtain a fourth feature of the query vector, the first processing including channel-wise convolution and point-wise convolution; perform a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, the second processing including channel-wise convolution; and perform a third processing on the fifth feature to obtain a third feature of the training image.

[0044] In one possible implementation, the model to be trained is used to: perform channel-wise convolution and point-wise convolution on a pre-set query vector to obtain the sixth feature of the query vector; and add the sixth feature and the query vector to obtain the fourth feature of the query vector.

[0045] In one possible implementation, the model to be trained is used to: upsample the fourth feature to obtain the seventh feature of the query vector; fuse the second and seventh features to obtain the eighth feature of the training image; perform channel-wise convolution on the eighth feature to obtain the ninth feature of the training image; and add the ninth and seventh features to obtain the fifth feature of the training image.

[0046] In one possible implementation, the model to be trained is used to: process the fifth feature based on a feedforward neural network to obtain the tenth feature of the training image; add the fifth feature and the tenth feature to obtain the eleventh feature of the training image; and pool the eleventh feature to obtain the third feature of the training image.

[0047] In one possible implementation, channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

[0048] In one possible implementation, pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

[0049] In one possible implementation, the query vector contains multiple parameters, the number of which is related to the number of objects.

[0050] A fifth aspect of this application provides a target detection apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the target detection apparatus performs the method described in the first aspect or any possible implementation thereof.

[0051] A sixth aspect of this application provides a model training apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training apparatus performs the method described in the second aspect or any possible implementation thereof.

[0052] A seventh aspect of this application provides a circuit system including a processing circuit configured to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0053] An eighth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory to cause the processor to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0054] In one possible implementation, the processor is coupled to the memory via an interface.

[0055] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0056] A ninth aspect of this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0057] A tenth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0058] In this embodiment, when target detection is required on a target image, a target image containing the object to be detected can be acquired first and input into the target model. Next, the target model can extract features from the target image to obtain a first feature. Then, the target model can encode the first feature to obtain a second feature. Subsequently, the target model can decode the second feature based on a preset query vector to obtain a third feature. Finally, the target model can obtain the detection result of the target image based on the third feature. The detection result can be used to determine the object's location information and category. Thus, target detection for the target model is completed. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations involve at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection, enabling the target detection task to be completed quickly even on devices with lower computing power, improving the efficiency of target detection and thus enhancing the user experience. Attached Figure Description

[0059] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0060] Figure 2a A schematic diagram of the target detection system provided in the embodiments of this application;

[0061] Figure 2b This is another structural schematic diagram of the target detection system provided in the embodiments of this application;

[0062] Figure 2c A schematic diagram of the target detection equipment provided in the embodiments of this application;

[0063] Figure 3 A schematic diagram of the system 100 architecture provided in the embodiments of this application;

[0064] Figure 4 A schematic diagram of the structure of the target model provided in the embodiments of this application;

[0065] Figure 5 A schematic flowchart of the target detection method provided in the embodiments of this application;

[0066] Figure 6 A schematic diagram of the structure of the target model provided in the embodiments of this application;

[0067] Figure 7 A schematic diagram of the decoder provided in an embodiment of this application;

[0068] Figure 8 Another structural schematic diagram of the target model provided in the embodiments of this application;

[0069] Figure 9 A schematic diagram of the comparison results provided for an embodiment of this application;

[0070] Figure 10 A schematic flowchart of the model training method provided in the embodiments of this application;

[0071] Figure 11 A schematic diagram of the target detection device provided in the embodiments of this application;

[0072] Figure 12 A schematic diagram of the structure of the model training apparatus provided in the embodiments of this application;

[0073] Figure 13 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0074] Figure 14 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0075] Figure 15 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0076] This application provides a target detection method and related equipment, which can quickly complete target detection tasks, improve the efficiency of target detection tasks, and thus improve user experience.

[0077] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0078] Object detection is one of the most fundamental computer vision tasks and is crucial for many practical applications. In recent years, the transformer model and its variants have demonstrated outstanding performance in image classification tasks, so they can be transferred to object detection tasks to improve the processing results.

[0079] Currently, when it's necessary to detect objects within a target image, the target image can be first input into a transformer model. The transformer model can then extract initial features from the target image. Next, it can process these initial features using a self-attention mechanism to obtain intermediate features. Then, it can process these intermediate features using both self-attention and cross-attention mechanisms to obtain the final features. Finally, the transformer model can use these final features to obtain the detection results, thus determining the object's location and category. For example, if an image contains multiple objects such as a cat, grass, flowers, and trees, it can be input into a transformer model for processing. Based on these detection results, the location and category of each object within the image can be determined.

[0080] In the process described above, since the transformer model is mainly built on the attention mechanism, it requires a lot of computational cost to perform object detection. If the computing power of the device equipped with the transformer model is low, the object detection speed will be slow, affecting the efficiency of object detection and resulting in a poor user experience.

[0081] To address the aforementioned problems, this application provides a target detection method that can be implemented in conjunction with artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Using artificial intelligence for data processing is a common application of AI.

[0082] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1This is a structural diagram illustrating the main framework of artificial intelligence. The following explanation of the AI ​​framework is based on two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0083] (1) Infrastructure

[0084] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0085] (2) Data

[0086] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0087] (3) Data processing

[0088] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0089] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0090] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0091] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0092] (4) General ability

[0093] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0094] (5) Smart Products and Industry Applications

[0095] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0096] The following section introduces several application scenarios for this application.

[0097] Figure 2a This is a schematic diagram of a target detection system provided in an embodiment of this application. The target detection system includes a user device and a data processing device. The user device includes smart terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiator of the target detection request; typically, the request is initiated by the user through the user device.

[0098] The aforementioned data processing equipment can be cloud servers, network servers, application servers, management servers, or other devices or servers with data processing capabilities. The data processing equipment receives target detection requests from smart terminals through an interactive interface, and then performs target detection processing through a storage device for storing data and a data processing processor, employing methods such as machine learning, deep learning, search, inference, and decision-making. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0099] exist Figure 2aIn the target detection system shown, the user equipment can receive user instructions. For example, the user equipment can acquire an image input / selected by the user and then send a request to the data processing device. This causes the data processing device to perform target detection processing on the image on the user equipment, thereby obtaining the target detection result for the image. This allows the determination of the location information and categories of several objects in the image. For instance, the user can input or select a target image containing objects to be detected on the user equipment. The user equipment then sends a target detection request to the data processing device for the target image, causing the data processing device to perform target detection processing on the target image and obtain the target detection result. Based on the target detection result, the location information and category of the objects to be detected can be determined.

[0100] exist Figure 2a In this context, the data processing device can execute the target detection method of the embodiments of this application.

[0101] Figure 2b This is another structural schematic diagram of the target detection system provided in the embodiments of this application. Figure 2b In this context, the user equipment (UE) directly functions as a data processing device. After determining an image input / selected by the user, the UE can directly perform target detection processing on that image. The specific process is similar to... Figure 2a Similar to the description above, it will not be repeated here.

[0102] exist Figure 2b In the target detection system shown, the user equipment can receive instructions from the user. For example, the user can input or select a target image containing the object to be detected on the user equipment. Then, the user equipment performs target detection processing on the target image to obtain the target detection result of the target image. Therefore, the location information and category of the object to be detected can be determined based on the target detection result.

[0103] exist Figure 2b In this context, the user equipment itself can execute the target detection method of the embodiments of this application.

[0104] Figure 2c This is a schematic diagram of a target detection device provided in an embodiment of this application.

[0105] The above Figure 2a and Figure 2b The user equipment in the context can specifically be Figure 2c Local device 301 or local device 302 in the system. Figure 2a The data processing equipment in the middle can specifically be Figure 2cThe execution device 210 in the process includes a data storage system 250 that can store the data to be processed by the execution device 210. The data storage system 250 can be integrated into the execution device 210 or set up in the cloud or on other network servers.

[0106] Figure 2a and Figure 2b The processor in the system can perform data training / machine learning / deep learning using neural network models or other models (e.g., support vector machine-based models), and then use the data to train or learn the model to perform object detection processing on the image, thereby obtaining the corresponding object detection results.

[0107] Figure 3 A schematic diagram of the system 100 architecture provided in this application embodiment, in Figure 3 In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data in this embodiment may include various scheduled tasks, callable resources, and other parameters.

[0108] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0109] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0110] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These target models / rules can then be used to achieve the aforementioned objectives or complete the aforementioned tasks, thereby providing the user with the required results. The training data can be stored in the database 130 and originates from training samples collected by the data acquisition device 160.

[0111] exist Figure 3In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0112] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this context, the data storage system 150 is an external memory relative to the execution device 110. However, in other cases, the data storage system 150 can also be placed within the execution device 110. For example... Figure 3 As shown, a neural network can be trained using training device 120.

[0113] This application also provides a chip including a neural network processor (NPU). This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rules.

[0114] The Neural Processing Unit (NPU) is a coprocessor mounted on the main central processing unit (CPU) (host CPU), where tasks are assigned by the CPU. The core of the NPU is the computation circuitry, which is controlled by a controller to retrieve data from memory (weight memory or input memory) and perform calculations.

[0115] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0116] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory and caches it in each PE (Process Equipment) of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator.

[0117] Vector computation units can further process the output of computational circuits, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, vector computation units can be used for computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0118] In some implementations, the vector computation unit can store the processed output vector into a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, for example, for use in subsequent layers of a neural network.

[0119] The unified memory is used to store input data and output data.

[0120] The weight data is directly transferred from the external memory to the input memory and / or unified memory, stored in the weight memory, and stored in the unified memory to the external memory through the direct memory access controller (DMAC).

[0121] The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC, and instruction fetch memory via a bus.

[0122] The instruction fetch buffer, connected to the controller, is used to store the instructions used by the controller.

[0123] The controller is used to invoke instructions cached in the memory to control the operation of the computing accelerator.

[0124] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, while external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDRSDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0125] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0126] (1) Neural Network

[0127] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0128]

[0129] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0130] The work of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by Wx, operation 4 by +b, and operation 5 by a(). The term "space" is used here because the objects being classified are not individual things, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight vector, and each value in this vector represents the weight value of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above; that is, the weights W of each layer control how the space is transformed. The purpose of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control the transformation space, and more specifically, learning the weight matrix.

[0131] Because the goal is for the neural network's output to be as close as possible to the actual predicted value, the network compares its current prediction with the desired target value. Based on the difference, the weight vector of each layer is updated (usually before the first update, where parameters are pre-configured for each layer). For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network can predict the desired target value. Therefore, it's necessary to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the neural network becomes a process of minimizing this loss.

[0132] (2) Backpropagation algorithm

[0133] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0134] The method provided in this application is described below from the perspectives of neural network training and neural network application.

[0135] The model training method provided in this application involves data sequence processing and can be applied to data training, machine learning, deep learning, and other methods. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (e.g., training images in the model training method provided in this application), ultimately obtaining a trained neural network (such as the target model in the model training method provided in this application). Furthermore, the target detection method provided in this application can utilize the trained neural network, inputting input data (e.g., target images in the target detection method provided in this application) into the trained neural network to obtain output data (such as the detection results of the target images in the target detection method provided in this application). It should be noted that the model training method and the target detection method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of a whole process: such as the model training stage and the model application stage.

[0136] The target detection method provided in this application can be implemented using a target model. Figure 4 A schematic diagram of the structure of the target model provided in the embodiments of this application, such as Figure 4 As shown, the target model includes multiple modules such as a backbone network, encoder, decoder, and detection network (detection head), all of which are pre-trained. The input terminals of the backbone network and the first input terminal of the decoder serve as the input terminals of the entire target model. The output terminal of the backbone network is connected to the input terminal of the encoder, the output terminal of the encoder is connected to the second input terminal of the decoder, the output terminal of the decoder is connected to the input terminal of the detection network, and the output terminal of the detection network serves as the output terminal of the entire target model. To understand the workflow of object detection based on the target model, the following section combines... Figure 5 This workflow will be described in more detail. Figure 5 A flowchart illustrating the target detection method provided in this application embodiment is shown below. Figure 5 As shown, the method includes:

[0137] 501. Obtain the target image, which contains the object to be detected.

[0138] In this embodiment, when object detection is required on a target image, the target image can be acquired first. The target image contains one or more objects to be detected. For example, if object detection is required on a certain image, the image can be acquired first, and this image may contain multiple objects such as cats, flowers, grass, trees, the ground, and walls to be detected.

[0139] 502. Perform feature extraction on the target image to obtain the first feature of the target image.

[0140] After obtaining the target image, it can be input into the target model to extract features from the target image, thereby obtaining the first feature of the target image.

[0141] Specifically, the target model can obtain the first feature of the target image in the following way:

[0142] After obtaining the target image, it can be input into the backbone network of the target model. Upon receiving the target image, the backbone network can perform a series of feature extractions on the target image to obtain the first feature of the target image, and then send the first feature of the target image to the encoder.

[0143] For example, such as Figure 6 As shown ( Figure 6 (This is a schematic diagram of the target model provided in an embodiment of this application). Suppose the target image contains a cat, which is the object to be detected. The target image is input into the backbone network of the target model. The backbone network can extract features from the target image to obtain the visual features of the target image (i.e., the aforementioned first feature), and send the visual features of the target image to the encoder.

[0144] 503. Encode the first feature to obtain the second feature of the target image. The encoding includes at least one convolution and does not include attention-based processing.

[0145] After obtaining the first feature of the target image, the target model can encode the first feature to obtain the second feature of the target image. It is worth noting that the encoding of the first feature of the target image by the target model may include at least one convolution, but does not include any attention-based processing (e.g., self-attention-based processing, cross-attention-based processing, etc.).

[0146] Specifically, the target model can obtain the second feature of the target image in the following way:

[0147] After obtaining the first feature of the target image, the encoder can encode the first feature to obtain the second feature of the target image, and send the second feature to the decoder. Since the encoder contains at least one convolutional layer, such as channel-wise convolutional layers and pointwise convolutional layers, etc., the encoder can perform at least one channel-wise convolution and / or at least one pointwise convolution on the first feature of the target image to obtain the second feature of the target image.

[0148] Continuing with the example above, after obtaining the visual features of the target image, the encoder, composed of convolutional layers, can convolve these features, such as through channel-wise convolution and / or pointwise convolution, thereby enhancing the features and obtaining the encoded features of the target image (i.e., the aforementioned second feature). These encoded features are then sent to the decoder. The encoded features of the target image can also be understood as the enhanced visual features of the target image, which is beneficial for more effective target detection in subsequent steps. It should be noted that the size of the encoded features of the target image can be set to d×H×W, meaning the number of channels is d, the height is H, and the width is W. The size of the encoded features can be the same as or different from the size of the target image; no restriction is placed here.

[0149] More specifically, the channel-wise convolutional layers contained in the encoder can be represented in the following form:

[0150] The channel-wise convolutional layer used to construct the encoder can be one or more of the following: channel-wise standard convolutional layer, channel-wise deformable convolutional layer, and channel-wise dynamic convolutional layer. Correspondingly, the encoding of the first feature of the target image by the encoder can include one or more of the following: channel-wise standard convolution, channel-wise deformable convolution, and channel-wise dynamic convolution.

[0151] More specifically, the pointwise convolutional layers contained in the encoder can be represented in the following form:

[0152] The pointwise convolutional layer used to construct the encoder can be one or more of pointwise standard convolutional layers, pointwise deformable convolutional layers, and pointwise dynamic convolutional layers. Correspondingly, the encoding of the first feature of the target image by the encoder can include one or more of pointwise standard convolution, pointwise deformable convolution, and pointwise dynamic convolution.

[0153] 504. The second feature is decoded based on the preset query vector to obtain the third feature of the target image. The decoding includes at least one convolution and does not include attention-based processing.

[0154] When acquiring the target image, a preset query vector can also be obtained, and the target image and the preset query vector are input into the target model. The query vector contains multiple randomly generated parameters (or multiple parameters with fixed values). The number of parameters (which can be set according to actual needs and is not limited here) is related to the number of objects recognized by the target model. In other words, the number of parameters determines the number of objects recognized by the target model.

[0155] After obtaining the second feature of the target image, the target model can use a pre-defined query vector to decode the second feature to obtain the third feature of the target image. It is worth noting that the decoding of the second feature by the target model may include at least one convolution, but it does not include any attention-based processing.

[0156] Specifically, the target model can obtain the third feature of the target image in the following ways:

[0157] (1) After obtaining the preset query vector and the second feature of the target image, since the decoder contains three modules—a self-interaction module (SIM), a cross-interaction module (CIM), and a post-processing module—the SIM can first perform a first processing on the preset query vector to obtain the fourth feature of the query vector, and then send the fourth feature of the query vector to the CIM. The first processing performed by the SIM on the query vector may include at least channel-wise convolution and point-wise convolution, but does not include any attention-based processing.

[0158] (2) After obtaining the fourth feature of the query vector, the cross-interaction module can perform a second processing on the second feature of the target image and the fourth feature of the query vector to obtain the fifth feature of the target image, and send the fifth feature of the target image to the post-processing module. The second processing performed by the cross-interaction module on the second and fourth features may include at least channel-wise convolution, but does not include any attention-based processing.

[0159] (3) After obtaining the fifth feature of the target image, the post-processing module can perform third processing on the fifth feature of the target image to obtain the third feature of the target image and send the third feature of the target image to the detection network.

[0160] More specifically, the self-interactive module can obtain the fourth feature of the query vector in the following ways:

[0161] (1.1) After obtaining the preset query vector, since the self-interaction module contains at least one convolutional layer and an additive layer (skip-connected layer), for example, these convolutional layers can be channel-wise convolutional layers and pointwise convolutional layers, etc., the self-interaction module can first perform at least one channel-wise convolution and at least one pointwise convolution on the preset query vector to obtain the sixth feature of the query vector.

[0162] (1.2) After obtaining the sixth feature of the query vector, the self-interaction module can also add the sixth feature of the query vector and the query vector together to obtain the fourth feature of the query vector, and send the fourth feature of the query vector to the cross-interaction module.

[0163] For example, such as Figure 7 As shown ( Figure 7 A schematic diagram of the decoder provided in an embodiment of this application. Figure 7 Is Figure 6 Based on the above, the query vector is drawn as a set of randomly initialized parameters. The size of the query vector is d×N×M, where N×M determines the number of detection boxes output by the detection network (one detection box represents the position information of an object, so the number of detection boxes is also the number of objects). The values ​​of N and M can be the same or different, for example, N=10, M=20, etc.

[0164] The decoder comprises a SIM module, a CIM module, and a post-processing module. The SIM module consists of a sequentially cascaded channel-wise convolutional (k×kdwconv) layer, two pointwise convolutional (1×1conv) layers, and an additive layer. After a preset query vector is input to the SIM module of the decoder, the SIM module performs one channel-wise convolution and two pointwise convolutions on the query vector to obtain the initial self-interaction feature of the query vector (i.e., the aforementioned sixth feature). Next, the SIM module adds the query vector to the initial self-interaction feature of the query vector to obtain the final self-interaction feature of the query vector (i.e., the aforementioned fourth feature), and then sends the final self-interaction feature of the query vector to the CIM module.

[0165] More specifically, the cross-interaction module can obtain the fifth feature of the target image in the following ways:

[0166] (2.1) After obtaining the fourth feature of the query vector, since the cross-interaction module includes an upsampling layer, a fusion layer, a channel-wise convolutional layer, and an addition layer, the cross-interaction module can first upsample the fourth feature of the query vector to obtain the seventh feature of the query vector.

[0167] (2.2) After obtaining the seventh feature of the query vector and the second feature of the target image, the cross-interaction module can fuse the second feature of the target image and the seventh feature of the query vector (e.g., add, multiply, subtract, and splice, etc.) to obtain the eighth feature of the target image.

[0168] (2.3) After obtaining the eighth feature of the target image, the cross-interaction module can perform at least one channel-wise convolution on the eighth feature of the target image to obtain the ninth feature of the target image.

[0169] (2.4) After obtaining the ninth feature of the target image, the cross-interaction module can add the ninth feature of the target image and the seventh feature of the query vector to obtain the fifth feature of the target image, and send the fifth feature of the target image to the post-processing module. It can be seen that through steps (2.2) and (2.4), the cross-interaction module can enable sufficient interaction between the features of the target image and the features of the query vector, which is beneficial for better target detection in the subsequent process.

[0170] As in the example above, the CIM module consists of sequentially connected upsampling layers, fusion layers, channel-wise convolutional (k×kdwconv) layers, and summing layers. After obtaining the encoded features of the target image and the final self-interaction features of the query vector, the CIM module first upsamples the final self-interaction features of the query vector to obtain the upsampled self-interaction features of the query vector (i.e., the aforementioned seventh feature). The upsampling operation of the CIM module ensures that the size of the upsampled self-interaction features of the query vector is the same as the size of the encoded features of the target image, that is, the size of the upsampled self-interaction features of the query vector is also d×H×W. The upsampling operation performed by the CIM module is shown in the following formula:

[0171]

[0172] In the above formula, o represents the final self-interaction feature of the query vector. The self-interactive features are upsampled versions of the query vector.

[0173] Next, the CIM module fuses the upsampled self-interaction features of the query vector with the encoded features of the target image to obtain the fused features of the target image (i.e., the aforementioned eighth feature). Then, it performs channel-wise convolution on the fused features of the target image to obtain the depth features of the target image (i.e., the aforementioned ninth feature). Finally, it adds the depth features of the target image to the upsampled self-interaction features of the query vector to obtain the cross-interaction features of the target image (i.e., the aforementioned fifth feature), and sends the cross-interaction features of the target image to the post-processing module. This process is illustrated by the following formula:

[0174]

[0175] In the above formula, z represents the encoded features of the target image. The fusion features of the target image For the depth features of the target image, The cross-interaction features of the target image.

[0176] More specifically, the post-processing module can obtain the third feature of the target image in the following ways:

[0177] (3.1) After obtaining the fifth feature of the target image, since the post-processing module includes a feedforward neural network layer, an addition layer, and a pooling layer, the post-processing module can first process the fifth feature based on the feedforward neural network (e.g., convolution and / or fully connected layers, etc.) to obtain the tenth feature of the target image.

[0178] (3.2) After obtaining the tenth feature of the target image, the post-processing module can add the fifth feature and the tenth feature of the target image to obtain the eleventh feature of the target image.

[0179] (3.3) After obtaining the eleventh feature of the target image, the post-processing module can pool the eleventh feature of the target image to obtain the third feature of the target image, and send the third feature of the target image to the detection network.

[0180] As in the example above, the post-processing module includes a feedforward neural network (FFN) layer, an addition layer, and a pooling layer, all connected in series. After obtaining the cross-interaction features of the target image, the post-processing module processes these features to obtain the feedforward features (the aforementioned tenth feature). It then adds these feedforward features to the cross-interaction features to obtain the initial decoded features (the aforementioned eleventh feature). Finally, it pools these initial decoded features to obtain the final decoded features (the aforementioned third feature), and sends these final decoded features to the detection network. The size of this feature is d×N×M. This process is illustrated by the following formula:

[0181]

[0182] In the above formula, For feedforward features of the target image, These are the initial decoding features of the target image. These are the final decoded features of the target image.

[0183] 505. Based on the third feature, obtain the detection results of the target image. The detection results are used to determine the location information of the object and the category of the object.

[0184] After obtaining the third feature of the target image, the target model (detection network) can further process the third feature to obtain the detection result of the target image. It should be noted that the detection result includes the location information of at least one object detected by the model, the category of this at least one object, and the confidence level of this at least one object. Since the model not only identifies some necessary objects (e.g., foreground objects, such as cats, flowers, grass, trees, etc.) but also some unnecessary objects (e.g., background objects, such as the ground, walls, etc.), and unnecessary objects often have lower confidence levels, the location information and category of objects with lower confidence levels can be removed, while the location information and category of objects with higher confidence levels are retained as the final location information and category of the objects detected by the model. This completes the target detection for the target image.

[0185] Continuing with the example above, after obtaining the final decoded features of the target image, the detection network can classify and regress these features to obtain the final detection result. This result includes N×M detection boxes, the categories of the objects within each of the N×M detection boxes, and the confidence scores of the N×M detection boxes. Detection boxes with low confidence scores (e.g., below a certain threshold, the size of which can be set according to actual needs, without limitation here) can be removed, while the remaining detection boxes with higher confidence scores and the categories of the objects within these boxes can be retained. Thus, the target detection of the target image is successfully completed.

[0186] It should be understood that this embodiment is only illustrative of a target model containing one decoder. In practical applications, the target model may also contain multiple decoders in series, such as... Figure 8 As shown ( Figure 8 (This is another structural diagram of the target model provided in an embodiment of this application). Multiple decoders are arranged between the encoder and the detection network. The input of the first decoder is the output of the encoder and a preset query vector. The input of the second decoder is the output of the encoder, the output of the first decoder (i.e., the third feature), and the preset query vector (the input of the SIM module of the second decoder is the output of the first decoder and the preset query vector), ..., and the input of the last decoder is the output of the encoder, the output of the penultimate decoder, and the preset query vector. Then, the detection network can obtain the detection result of the target image based on the output of the last decoder.

[0187] Furthermore, the target model provided in the embodiments of this application (e.g., DECO in Table 1) can be compared with models of related technologies (e.g., the other models in Table 1 excluding DECO, such as FCOS, DETR, etc.) in terms of detection metrics (AP) and inference speed (FPS). The comparison results are shown in Table 1.

[0188] Table 1

[0189] Model Backbone GFLOPs FPS AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AP S ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> FasterR-CNN R50-FPN 180 26 40.2 61.0 43.8 24.2 43.5 52.0 FasterR-CNN R101-FPN 246 20 42.0 62.5 45.9 25.2 45.6 54.6 FCOS R50-FPN 201 23 38.7 57.4 41.8 22.9 42.5 50.1 FCOS R101-FPN 277 19 39.1 58.3 42.1 22.7 43.3 50.3 RetinaNet R50-FPN 239 21 37.4 56.7 39.6 20.0 40.7 49.7 RetinaNet R101-FPN 315 17 38.5 57.6 41.0 21.7 42.8 50.4 SparseR-CNN R50-FPN 150 20 37.9 56.0 40.5 20.7 40.0 53.5 OneNet-RetinaNet R50-FPN - 21 37.5 55.4 40.7 21.5 40.5 47.4 OneNet-FCOS R50-FPN - 26 38.9 57.2 42.2 23.9 41.8 49.4 DeFCN R50-FPN - 19 41.4 59.5 45.6 26.1 44.9 52.0 YOLOS-Ti DeiT-Tiny 21 52 28.7 47.2 28.9 9.7 29.2 46.0 YOLOS-S DeiT-Small 194 5 36.1 55.7 37.6 15.6 38.3 55.3 YOLOS-B DeiT-Base 538 2 42.0 62.2 44.4 19.5 45.3 62.1 DETR R34 88 34 31.6 47.6 33.3 13.3 34.1 49.1 DETR R50 97 28 39.5 60.3 41.4 17.5 43.0 59.1 DECO R50 103 35 37.8 57.9 40.3 17.8 42.5 53.6 DETR ConvNeXt-Tiny 104 25 42.1 63.6 44.3 18.8 45.5 62.8 DECO ConvNeXt-Tiny 110 28 41.3 62.0 43.7 20.5 45.8 59.6

[0190] Based on Table 1, the following can be obtained: Figure 9 The curve diagram shown ( Figure 9 (This is a schematic diagram of the comparison results provided in the embodiments of this application). It can be seen that the performance of the target model provided in the embodiments of this application is better than the performance of the model provided by related technologies.

[0191] In this embodiment, when target detection is required on a target image, a target image containing the object to be detected can be acquired first and input into the target model. Next, the target model can extract features from the target image to obtain a first feature. Then, the target model can encode the first feature to obtain a second feature. Subsequently, the target model can decode the second feature based on a preset query vector to obtain a third feature. Finally, the target model can obtain the detection result of the target image based on the third feature. The detection result can be used to determine the object's location information and category. Thus, target detection for the target model is completed. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations involve at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection, enabling the target detection task to be completed quickly even on devices with lower computing power, improving the efficiency of target detection and thus enhancing the user experience.

[0192] The above is a detailed description of the target detection method provided in the embodiments of this application. The model training method provided in the embodiments of this application will be introduced below. Figure 10 A schematic flowchart of the model training method provided in the embodiments of this application is shown below. Figure 10 As shown, the method includes:

[0193] 1001. Obtain training images containing the objects to be detected.

[0194] In this embodiment, when the model to be trained needs to be trained, a batch of training data can be obtained first, which includes training images. It should be noted that the true detection results of the training images are known, and the true detection results of the training images include the true location information of at least one object to be detected, the true category of the at least one object, and the true confidence level of the at least one object.

[0195] 1002. The training image is processed by the model to be trained to obtain the detection result of the training image. The detection result is used to determine the location information and category of the object. The model to be trained is used to: extract features from the training image to obtain the first feature of the training image; encode the first feature to obtain the second feature of the training image, wherein the encoding includes at least one convolution and does not include attention-based processing; decode the second feature based on a preset query vector to obtain the third feature of the training image, wherein the decoding includes at least one convolution and does not include attention-based processing; and obtain the detection result of the training image based on the third feature.

[0196] After obtaining the training image, it can be input into the model to be trained. The model processes the training image to obtain the (predicted) detection result. The detection result includes the (predicted) location information, the (predicted) category, and the (predicted) confidence level of at least one object. Specifically, the model to be trained is used to: extract features from the training image to obtain a first feature; encode the first feature to obtain a second feature, where the encoding includes at least one convolution and does not include attention-based processing; decode the second feature based on a preset query vector to obtain a third feature, where the decoding includes at least one convolution and does not include attention-based processing; and obtain the detection result of the training image based on the third feature.

[0197] In one possible implementation, the encoding includes at least one of the following: channel-wise convolution or point-wise convolution.

[0198] In one possible implementation, decoding the second feature based on a preset query vector to obtain the third feature of the training image includes: performing a first processing on the preset query vector to obtain a fourth feature of the query vector, the first processing including channel-wise convolution and point-wise convolution; performing a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, the second processing including channel-wise convolution; and performing a third processing on the fifth feature to obtain the third feature of the training image.

[0199] In one possible implementation, the first processing of the preset query vector to obtain the fourth feature of the query vector includes: performing channel-wise convolution and point-wise convolution on the preset query vector to obtain the sixth feature of the query vector; and adding the sixth feature and the query vector to obtain the fourth feature of the query vector.

[0200] In one possible implementation, the second processing of the second and fourth features to obtain the fifth feature of the training image includes: upsampling the fourth feature to obtain the seventh feature of the query vector; fusing the second and seventh features to obtain the eighth feature of the training image; performing channel-wise convolution on the eighth feature to obtain the ninth feature of the training image; and adding the ninth and seventh features to obtain the fifth feature of the training image.

[0201] In one possible implementation, the third processing of the fifth feature to obtain the third feature of the training image includes: processing the fifth feature based on a feedforward neural network to obtain the tenth feature of the training image; adding the fifth feature and the tenth feature to obtain the eleventh feature of the training image; and pooling the eleventh feature to obtain the third feature of the training image.

[0202] In one possible implementation, channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

[0203] In one possible implementation, pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

[0204] In one possible implementation, the query vector contains multiple parameters, the number of which is related to the number of objects.

[0205] It should be understood that the description of step 1002 can be used as a parameter. Figure 5 The relevant descriptions of steps 502 to 505 in the illustrated embodiment will not be repeated here.

[0206] 1003. Based on the detection results and the real detection results of the training images, train the model to be trained to obtain the target model.

[0207] After obtaining the detection results of the training images, since the actual detection results of the training images are known, the detection results of the training images and the actual detection results of the training images can be calculated using a preset loss function to obtain the target loss. The target loss is used to indicate the difference between the detection results of the training images and the actual detection results of the training images.

[0208] After obtaining the target loss, it can be used to update the parameters of the model to be trained, thus obtaining the updated model. The updated model can then be trained using the next batch of training data until the model training conditions are met (e.g., the target loss converges, etc.). Figure 5 The target model in the illustrated embodiment.

[0209] The target model trained in this embodiment has the function of object detection. Specifically, when object detection is required on a target image, a target image containing the object to be detected can be obtained first, and the target image can be input into the target model. Then, the target model can extract features from the target image to obtain the first feature of the target image. Then, the target model can encode the first feature of the target image to obtain the second feature of the target image. Subsequently, the target model can decode the second feature of the target image based on a preset query vector to obtain the third feature of the target image. Finally, the target model can obtain the detection result of the target image based on the third feature. The detection result can be used to determine the location information and category of the object. Thus, the object detection for the target model is completed. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost in the process of performing object detection. Even if the computing power of the device equipped with the target model is low, the object detection task can be completed quickly, improving the efficiency of the object detection task and thus improving the user experience.

[0210] The above is a detailed description of the target detection method and model training method provided in the embodiments of this application. The target detection device and model training device provided in the embodiments of this application will be described below. Figure 11 A schematic diagram of the target detection device provided in the embodiments of this application is shown below. Figure 11 As shown, the device includes a target model, and the device comprises:

[0211] The acquisition module 1101 is used to acquire a target image, which contains the object to be detected;

[0212] Extraction module 1102 is used to extract features from the target image to obtain the first feature of the target image;

[0213] The encoding module 1103 is used to encode the first feature to obtain the second feature of the target image. The encoding includes at least one convolution and does not include attention-based processing.

[0214] The decoding module 1104 is used to decode the second feature based on the preset query vector to obtain the third feature of the target image. The decoding includes at least one convolution and does not include attention-based processing.

[0215] The detection module 1105 is used to obtain the detection results of the target image based on the third feature. The detection results are used to determine the location information of the object and the category of the object.

[0216] In this embodiment, when target detection is required on a target image, a target image containing the object to be detected can be acquired first and input into the target model. Next, the target model can extract features from the target image to obtain a first feature. Then, the target model can encode the first feature to obtain a second feature. Subsequently, the target model can decode the second feature based on a preset query vector to obtain a third feature. Finally, the target model can obtain the detection result of the target image based on the third feature. The detection result can be used to determine the object's location information and category. Thus, target detection for the target model is completed. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations involve at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost during target detection, enabling the target detection task to be completed quickly even on devices with lower computing power, improving the efficiency of target detection and thus enhancing the user experience.

[0217] In one possible implementation, the encoding includes at least one of the following: channel-wise convolution or point-wise convolution.

[0218] In one possible implementation, the decoding module 1104 is configured to: perform a first processing on a preset query vector to obtain a fourth feature of the query vector, the first processing including channel-wise convolution and point-wise convolution; perform a second processing on the second feature and the fourth feature to obtain a fifth feature of the target image, the second processing including channel-wise convolution; and perform a third processing on the fifth feature to obtain a third feature of the target image.

[0219] In one possible implementation, the decoding module 1104 is used to: perform channel-wise convolution and point-wise convolution on a preset query vector to obtain a sixth feature of the query vector; and add the sixth feature and the query vector to obtain a fourth feature of the query vector.

[0220] In one possible implementation, the decoding module 1104 is used to: upsample the fourth feature to obtain the seventh feature of the query vector; fuse the second feature and the seventh feature to obtain the eighth feature of the target image; perform channel-wise convolution on the eighth feature to obtain the ninth feature of the target image; and add the ninth feature and the seventh feature to obtain the fifth feature of the target image.

[0221] In one possible implementation, the decoding module 1104 is used to: process the fifth feature based on a feedforward neural network to obtain the tenth feature of the target image; add the fifth feature and the tenth feature to obtain the eleventh feature of the target image; and pool the eleventh feature to obtain the third feature of the target image.

[0222] In one possible implementation, channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

[0223] In one possible implementation, pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

[0224] In one possible implementation, the query vector contains multiple parameters, the number of which is related to the number of objects.

[0225] Figure 12 A schematic diagram of the model training apparatus provided in the embodiments of this application is shown below. Figure 12 As shown, the device includes:

[0226] The acquisition module 1201 is used to acquire training images, which contain objects to be detected;

[0227] The processing module 1202 is used to process the training image through the model to be trained to obtain the detection result of the training image. The detection result is used to determine the position information and category of the object. The model to be trained is used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, wherein the encoding includes at least one convolution and does not include attention-based processing; decode the second feature based on a preset query vector to obtain a third feature of the training image, wherein the decoding includes at least one convolution and does not include attention-based processing; and obtain the detection result of the training image based on the third feature.

[0228] Training module 1203 is used to train the model to be trained based on the detection results and the real detection results of the training images to obtain the target model.

[0229] The target model trained in this embodiment has the function of object detection. Specifically, when object detection is required on a target image, a target image containing the object to be detected can be obtained first, and the target image can be input into the target model. Then, the target model can extract features from the target image to obtain the first feature of the target image. Then, the target model can encode the first feature of the target image to obtain the second feature of the target image. Subsequently, the target model can decode the second feature of the target image based on a preset query vector to obtain the third feature of the target image. Finally, the target model can obtain the detection result of the target image based on the third feature. The detection result can be used to determine the location information and category of the object. Thus, the object detection for the target model is completed. In the aforementioned process, the main operations performed by the target model include encoding and decoding. Both encoding and decoding operations include at least one convolution and do not include any attention-based processing. This allows the target model to consume less computational cost in the process of performing object detection. Even if the computing power of the device equipped with the target model is low, the object detection task can be completed quickly, improving the efficiency of the object detection task and thus improving the user experience.

[0230] In one possible implementation, the encoding includes at least one of the following: channel-wise convolution or point-wise convolution.

[0231] In one possible implementation, the model to be trained is used to: perform a first processing on a preset query vector to obtain a fourth feature of the query vector, the first processing including channel-wise convolution and point-wise convolution; perform a second processing on the second feature and the fourth feature to obtain a fifth feature of the training image, the second processing including channel-wise convolution; and perform a third processing on the fifth feature to obtain a third feature of the training image.

[0232] In one possible implementation, the model to be trained is used to: perform channel-wise convolution and point-wise convolution on a pre-set query vector to obtain the sixth feature of the query vector; and add the sixth feature and the query vector to obtain the fourth feature of the query vector.

[0233] In one possible implementation, the model to be trained is used to: upsample the fourth feature to obtain the seventh feature of the query vector; fuse the second and seventh features to obtain the eighth feature of the training image; perform channel-wise convolution on the eighth feature to obtain the ninth feature of the training image; and add the ninth and seventh features to obtain the fifth feature of the training image.

[0234] In one possible implementation, the model to be trained is used to: process the fifth feature based on a feedforward neural network to obtain the tenth feature of the training image; add the fifth feature and the tenth feature to obtain the eleventh feature of the training image; and pool the eleventh feature to obtain the third feature of the training image.

[0235] In one possible implementation, channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

[0236] In one possible implementation, pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

[0237] In one possible implementation, the query vector contains multiple parameters, the number of which is related to the number of objects.

[0238] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0239] This application also relates to an execution device. Figure 13 This is a schematic diagram of the execution device provided in an embodiment of this application. Figure 13 As shown, the execution device 1300 can specifically manifest as a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited here. Among them, the execution device 1300 may deploy... Figure 11 The target detection device described in the corresponding embodiment is used to implement Figure 5 This corresponds to the target detection function in the embodiment. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303, and a memory 1304 (wherein the execution device 1300 may have one or more processors 1303). Figure 13 (Taking a processor as an example), processor 1303 may include application processor 13031 and communication processor 13032. In some embodiments of this application, receiver 1301, transmitter 1302, processor 1303 and memory 1304 may be connected via bus or other means.

[0240] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0241] Processor 1303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0242] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1303. The processor 1303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1303 or by instructions in software form. The processor 1303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1303 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1304. Processor 1303 reads the information in memory 1304 and, in conjunction with its hardware, completes the steps of the above method.

[0243] Receiver 1301 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1302 can be used to output digital or character information through the first interface; transmitter 1302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1302 may also include a display device such as a display screen.

[0244] In one embodiment of this application, the processor 1303 is used to... Figure 5 The target model in the corresponding embodiment completes target detection for the target image.

[0245] This application also relates to a training device. Figure 14 This is a schematic diagram of the structure of a training device provided in an embodiment of this application. Figure 14 As shown, the training device 1400 is implemented by one or more servers. The training device 1400 can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) 1414 (e.g., one or more processors) and memory 1432, and one or more storage media 1430 (e.g., one or more mass storage devices) for storing application programs 1442 or data 1444. The memory 1432 and storage media 1430 can be temporary or persistent storage. The program stored in the storage media 1430 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the training device. Furthermore, the CPU 1414 may be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the training device 1400.

[0246] The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458; or, one or more operating systems 1441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0247] Specifically, the training equipment can perform Figure 10 The model training method in the corresponding embodiment is used to obtain the target model.

[0248] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0249] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.

[0250] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0251] For details, please refer to Figure 15 , Figure 15 This is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU1500). The NPU1500 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1503, which is controlled by the controller 1504 to extract matrix data from the memory and perform multiplication operations.

[0252] In some implementations, the arithmetic circuit 1503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1503 is a two-dimensional pulsating array. The arithmetic circuit 1503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1503 is a general-purpose matrix processor.

[0253] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1508.

[0254] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 via Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 via DMAC.

[0255] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1509.

[0256] The Bus Interface Unit (BIU) 1513 is used by the instruction fetch memory 1509 to fetch instructions from external memory, and also by the memory access controller 1505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0257] The DMAC is mainly used to move input data from external memory DDR to unified memory 1506, or to weight data to weight memory 1502, or to input data to input memory 1501.

[0258] The vector computation unit 1507 includes multiple processing units that further process the output of the computation circuit 1503 when necessary, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as Batch Normalization, pixel-level summation, and upsampling of the predicted label plane.

[0259] In some implementations, the vector computation unit 1507 can store the processed output vector in the unified memory 1506. For example, the vector computation unit 1507 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1503, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1507 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1503, for example, for use in subsequent layers of the neural network.

[0260] The instruction fetch buffer 1509 connected to the controller 1504 is used to store the instructions used by the controller 1504;

[0261] Unified memory 1506, input memory 1501, weighted memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0262] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0263] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0265] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0266] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A target detection method, characterized in that, The method is implemented through a target model, and the method includes: Acquire a target image, wherein the target image contains the object to be detected; Feature extraction is performed on the target image to obtain the first feature of the target image; The first feature is encoded to obtain the second feature of the target image, wherein the encoding includes at least one convolution and does not include attention-based processing; The second feature is decoded based on a preset query vector to obtain the third feature of the target image. The decoding includes at least one convolution and does not include attention-based processing. Based on the third feature, the detection result of the target image is obtained, and the detection result is used to determine the location information of the object and the category of the object; The process of decoding the second feature based on a preset query vector to obtain the third feature of the target image includes: The preset query vector is subjected to a first process to obtain the fourth feature of the query vector. The first process includes channel-wise convolution and point-wise convolution. The second feature and the fourth feature are subjected to a second processing to obtain the fifth feature of the target image. The second processing includes channel-wise convolution. The fifth feature is processed in a third way to obtain the third feature of the target image.

2. The method according to claim 1, characterized in that, The encoding includes at least one of the following: channel-wise convolution or point-wise convolution.

3. The method according to claim 1, characterized in that, The first processing of the preset query vector to obtain the fourth feature of the query vector includes: The sixth feature of the query vector is obtained by performing channel-wise convolution and point-wise convolution on the preset query vector. The sixth feature and the query vector are added together to obtain the fourth feature of the query vector.

4. The method according to claim 1 or 3, characterized in that, The second processing of the second feature and the fourth feature to obtain the fifth feature of the target image includes: Upsample the fourth feature to obtain the seventh feature of the query vector; The second feature and the seventh feature are fused to obtain the eighth feature of the target image; The eighth feature is convolved channel by channel to obtain the ninth feature of the target image; The ninth feature and the seventh feature are added together to obtain the fifth feature of the target image.

5. The method according to any one of claims 1 to 3, characterized in that, The third processing of the fifth feature to obtain the third feature of the target image includes: The fifth feature is processed using a feedforward neural network to obtain the tenth feature of the target image; The fifth feature and the tenth feature are added together to obtain the eleventh feature of the target image; Pooling is performed on the eleventh feature to obtain the third feature of the target image.

6. The method according to any one of claims 1 to 3, characterized in that, The channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

7. The method according to any one of claims 1 to 3, characterized in that, The pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

8. The method according to any one of claims 1 to 3, characterized in that, The query vector contains multiple parameters, the number of which is related to the number of objects.

9. A model training method, characterized in that, The method includes: Acquire training images, which contain the objects to be detected; The training image is processed by the model to be trained to obtain the detection result of the training image. The detection result is used to determine the location information and category of the object. The model to be trained is used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, wherein the encoding includes at least one convolution and does not include attention-based processing; decode the second feature based on a preset query vector to obtain a third feature of the training image, wherein the decoding includes at least one convolution and does not include attention-based processing; and obtain the detection result of the training image based on the third feature. Based on the detection results and the actual detection results of the training images, the model to be trained is trained to obtain the target model; The process of decoding the second feature based on a preset query vector to obtain the third feature of the training image includes: The preset query vector is subjected to a first process to obtain the fourth feature of the query vector. The first process includes channel-wise convolution and point-wise convolution. The second feature and the fourth feature are subjected to a second processing to obtain the fifth feature of the training image. The second processing includes channel-wise convolution. The fifth feature is processed in a third way to obtain the third feature of the training image.

10. The method according to claim 9, characterized in that, The encoding includes at least one of the following: channel-wise convolution or point-wise convolution.

11. The method according to claim 9, characterized in that, The first processing of the preset query vector to obtain the fourth feature of the query vector includes: The sixth feature of the query vector is obtained by performing channel-wise convolution and point-wise convolution on the preset query vector. The sixth feature and the query vector are added together to obtain the fourth feature of the query vector.

12. The method according to claim 9 or 11, characterized in that, The second processing of the second feature and the fourth feature to obtain the fifth feature of the training image includes: Upsample the fourth feature to obtain the seventh feature of the query vector; The second feature and the seventh feature are fused to obtain the eighth feature of the training image; The eighth feature is convolved channel by channel to obtain the ninth feature of the training image; The ninth feature and the seventh feature are added together to obtain the fifth feature of the training image.

13. The method according to any one of claims 9 to 11, characterized in that, The third processing of the fifth feature to obtain the third feature of the training image includes: The fifth feature is processed using a feedforward neural network to obtain the tenth feature of the training image; The fifth feature and the tenth feature are added together to obtain the eleventh feature of the training image; Pooling is performed on the eleventh feature to obtain the third feature of the training image.

14. The method according to any one of claims 9 to 11, characterized in that, The channel-wise convolution includes at least one of the following: channel-wise standard convolution, channel-wise deformable convolution, or channel-wise dynamic convolution.

15. The method according to any one of claims 9 to 11, characterized in that, The pointwise convolution includes at least one of the following: pointwise standard convolution, pointwise deformable convolution, or pointwise dynamic convolution.

16. The method according to any one of claims 9 to 11, characterized in that, The query vector contains multiple parameters, the number of which is related to the number of objects.

17. A target detection device, characterized in that, The device includes a target model, and the device comprises: An acquisition module is used to acquire a target image, wherein the target image contains the object to be detected; The extraction module is used to extract features from the target image to obtain the first feature of the target image; An encoding module is used to encode the first feature to obtain a second feature of the target image, wherein the encoding includes at least one convolution and does not include attention-based processing. A decoding module is used to decode the second feature based on a preset query vector to obtain the third feature of the target image. The decoding includes at least one convolution and does not include attention-based processing. The detection module is used to obtain the detection result of the target image based on the third feature, and the detection result is used to determine the location information of the object and the category of the object; The decoding module is used for: The preset query vector is subjected to a first process to obtain the fourth feature of the query vector. The first process includes channel-wise convolution and point-wise convolution. The second feature and the fourth feature are subjected to a second processing to obtain the fifth feature of the target image. The second processing includes channel-wise convolution. The fifth feature is processed in a third way to obtain the third feature of the target image.

18. A model training device, characterized in that, The device includes: An acquisition module is used to acquire training images, wherein the training images contain objects to be detected; The processing module is used to process the training image using a model to be trained to obtain a detection result of the training image. The detection result is used to determine the location information and category of the object. The model to be trained is used to: extract features from the training image to obtain a first feature of the training image; encode the first feature to obtain a second feature of the training image, wherein the encoding includes at least one convolution and does not include attention-based processing; decode the second feature based on a preset query vector to obtain a third feature of the training image, wherein the decoding includes at least one convolution and does not include attention-based processing; and obtain the detection result of the training image based on the third feature. The training module is used to train the model to be trained based on the detection results and the real detection results of the training images to obtain the target model; The model to be trained is used for: The preset query vector is subjected to a first process to obtain the fourth feature of the query vector. The first process includes channel-wise convolution and point-wise convolution. The second feature and the fourth feature are subjected to a second processing to obtain the fifth feature of the training image. The second processing includes channel-wise convolution. The fifth feature is processed in a third way to obtain the third feature of the training image.

19. A target detection device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the target detection device performs the method as described in any one of claims 1 to 16.

20. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 16.

21. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 16.

Images