Data processing method and related apparatus

Abstract

Disclosed in the embodiments of the present application is a data processing method, which is applied to the field of artificial intelligence. The method comprises: acquiring an image; obtaining target feature information of the image by means of a target network, wherein the target feature information is a fusion result of first feature information and second feature information, the first feature information is obtained by means of a first sub-network performing target feature extraction on the image, the second feature information is obtained by means of a second sub-network performing target instance segmentation on the image, and the first sub-network and the second sub-network are two branch networks in the target network; and on the basis of the target feature information, obtaining target information by means of a prediction network, wherein the target information is used for indicating a target object in the image. In this way, advantageous features of different branches are extracted and fused by means of a parallel multi-branch model architecture, such that the advantageous features are enhanced in a complementary manner, thereby improving the feature expression capability, enhancing the overall performance of a target network, and reducing the false detection rate during image detection.

Description

A data processing method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202411001671.X, filed on July 24, 2024, entitled “A Data Processing Method and Related Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a data processing method and related apparatus. Background Technology

[0003] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0004] Crowd counting (CC) is an important task in computer vision. Its goal is to identify and locate individual heads in an image and assess crowd density, flow, and other information, making it widely used in public spaces. The detection models used to perform CC primarily distinguish features between the target object and the background in an image to achieve object identification and localization.

[0005] However, in practical applications, the quality of captured images is often reduced due to factors such as rapid movement of people in public areas and insufficient lighting. Faced with complex and ever-changing densely populated scenes, current detection models exhibit a high false detection rate. Summary of the Invention

[0006] This application provides a data processing method and related apparatus to enhance the robustness of detection models and reduce the false detection rate when detecting images.

[0007] In a first aspect, embodiments of this application provide a data processing method, the method comprising:

[0008] Acquire images;

[0009] The target network obtains the target feature information of the image. The target feature information is the fusion result of the first feature information and the second feature information. The first feature information is obtained by the first sub-network extracting the target features of the image, and the second feature information is obtained by the second sub-network segmenting the target instance of the image. The first sub-network and the second sub-network are two branch networks in the target network.

[0010] Based on the target feature information, a prediction network is used to obtain target information, which is used to indicate the target object in the image.

[0011] In this application, the input image is the image for which target object recognition is required. It can be a picture taken in the detection scene or a frame extracted from a video. The target object to be recognized can be a person, an animal (such as a cat, dog, bird, etc.), a vehicle, etc.; or it can be a part of the target object, such as a person's head or a vehicle's license plate.

[0012] In this application, the target network includes two parallel branch tasks: a first sub-network and a second sub-network. These two branch networks perform corresponding preference processing on image features based on their own network characteristics. The first sub-network is the detection branch of the target network, and the second sub-network is the instance segmentation branch of the target network. The detection branch in this application is equivalent to two stages: the first stage extracts target features from the input content, and the second stage predicts and infers from the feature information. The step of fusing the first and second feature information to obtain the target feature information of the image is embedded between these two stages, providing support for the entire detection process.

[0013] By using the above method, the advantageous features of different branches are extracted and integrated, so that they complement and enhance each other, thereby improving the expressive power of the features, enhancing the overall performance of the target network, and reducing the false detection rate when detecting images.

[0014] In one possible implementation, target feature information of the image is obtained through a target network, including:

[0015] The basic image features of an image are obtained through a feature extraction network, which is the backbone network of the target network.

[0016] The basic image features are input into the first sub-network and the second sub-network respectively to obtain the first feature information and the second feature information;

[0017] Based on the basic image features, the first feature information and the second feature information are fused to obtain the target feature information.

[0018] In this application, the target network can be divided into two parts. The first part is the original backbone network of the target network, whose core function is to extract basic image features. The second part is a parallel branch network structure, and the input parameters of each branch network are the basic image features extracted in the first part.

[0019] In one possible implementation, based on basic image features, the first feature information and the second feature information are fused to obtain target feature information, including:

[0020] A global representation of the image is obtained based on the extraction of basic image features;

[0021] The target feature information is obtained by fusing the first and second feature information through global representation.

[0022] In this application, multi-head attention fusion is employed to perform deep processing on basic image features to obtain a global representation of the image, such as overall attributes like contextual information, shape characteristics, and texture details. Then, a feature fusion module guided by the global representation is used to achieve complementary fusion of feature information from the two branches. For each point of the features from the two branches, two vectors can be obtained. By introducing the global representation, dynamic fusion can be achieved among these three vectors through an attention mechanism, resulting in a more comprehensive and expressive feature representation.

[0023] In one possible implementation, before obtaining the basic image features of the image through a feature extraction network, the method further includes:

[0024] The feature extraction network is updated using the second feature information.

[0025] In this application, the feature extraction network can be pre-trained by obtaining feature information from the instance segmentation network, thereby improving the instance perception capability of the feature extraction network and enhancing the accuracy and expressive power of its output basic image features.

[0026] In one possible implementation, before obtaining the target information through a prediction network based on the target feature information, the method further includes:

[0027] Based on the second feature information, a first position distribution is obtained, which is used to indicate the position information of at least one target object in the image;

[0028] Based on the first position distribution, multiple threshold values ​​are obtained; wherein, the first threshold value is used to indicate the confidence threshold or overlap threshold of the output result of the prediction model in the first image region, the confidence threshold or overlap threshold is related to the density of the target object in the first image region, the first prediction information is one of the multiple prediction information, and the first image region is a region in the image;

[0029] Based on the target feature information, a prediction network is used to obtain target information, including:

[0030] Based on target feature information and multiple threshold values, target information is obtained through a prediction network.

[0031] In this application, the fitting target of the instance segmentation network (second sub-network) can be an instance segmentation map or an independent instance map, which can identify the target objects in the map. That is, this process provides an intuitive representation of the location distribution of target objects in the image. By dividing the image into multiple regions and combining the number (density) of target objects in each region of the binarized image, the corresponding thresholds for each region can be dynamically adjusted during the prediction process, including but not limited to the threshold used to verify the confidence of the detection boxes, and the overlap threshold required to calculate the intersection-union ratio of the detection boxes.

[0032] The above method improves the flexibility of detection box result processing and the accuracy and robustness of NMS post-processing.

[0033] In one possible implementation, the first sub-network includes at least one of the following: a density map regression network, a point localization network, or an object detection network.

[0034] Secondly, embodiments of this application provide a data processing method, the method comprising:

[0035] Acquire images;

[0036] The basic image features of the image are obtained through a feature extraction network. The feature extraction network is a network updated based on the second feature information. The second feature information is the feature information obtained by the instance segmentation network to segment the target instance of the low-quality image.

[0037] Based on the basic image features, target information is obtained through a detection network. The target information is used to indicate the target object in the image, and the detection network is used to extract the target features of the target object.

[0038] In this application, similar to the first aspect mentioned above, the entire detection model can also be regarded as two parts. The first part is the original backbone network of the target network, whose core function is to extract basic image features. This backbone network is a network fine-tuned and trained by the instance segmentation network. The second part is the detection network, which is used to extract target features from the basic image features obtained in the first part.

[0039] Using the above method, the basic image features on which prediction and inference rely can be regarded as a comprehensive utilization of the features extracted by the object detection network and the features extracted by the instance segmentation network, which can improve the overall performance and accuracy of the model.

[0040] In one possible implementation, the detection network includes at least one of the following: a density map regression network, a point localization network, or an object detection network.

[0041] Thirdly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0042] The acquisition module is used to acquire images;

[0043] The processing module is used to obtain the target feature information of the image through the target network. The target feature information is the fusion result of the first feature information and the second feature information. The first feature information is obtained by the first sub-network extracting the target features of the image, and the second feature information is obtained by the second sub-network segmenting the target instance of the image. The first sub-network and the second sub-network are two branch networks in the target network.

[0044] The processing module is also used to obtain target information based on target feature information through a prediction network. The target information is used to indicate the target object in the image.

[0045] In one possible implementation, the processing module is specifically used for:

[0046] The basic image features of an image are obtained through a feature extraction network, which is the backbone network of the target network.

[0047] The basic image features are input into the first sub-network and the second sub-network respectively to obtain the first feature information and the second feature information;

[0048] Based on the basic image features, the first feature information and the second feature information are fused to obtain the target feature information.

[0049] In one possible implementation, the processing module is specifically used to: extract a global representation of the image based on the basic image features;

[0050] The target feature information is obtained by fusing the first and second feature information through global representation.

[0051] In one possible implementation, before obtaining the basic image features of the image through the feature extraction network, the processing module is further used for:

[0052] The feature extraction network is updated using the second feature information.

[0053] In one possible implementation, before obtaining the target information through a prediction network based on the target feature information, the processing module is further used for:

[0054] Based on the second feature information, a first position distribution is obtained, which is used to indicate the position information of at least one target object in the image;

[0055] Based on the first position distribution, multiple threshold values ​​are obtained; wherein, the first threshold value is used to indicate the confidence threshold or overlap threshold of the output result of the prediction model in the first image region, the confidence threshold or overlap threshold is related to the density of the target object in the first image region, the first prediction information is one of the multiple prediction information, and the first image region is a region in the image;

[0056] The processing module is specifically used to obtain target information through a prediction network based on target feature information and multiple threshold values.

[0057] In one possible implementation, the first sub-network includes at least one of the following: a density map regression network, a point localization network, or an object detection network.

[0058] Fourthly, embodiments of this application provide a data processing apparatus, which includes:

[0059] The acquisition module is used to acquire images;

[0060] The processing module is used to obtain the basic image features of the image through the feature extraction network. The feature extraction network is a network updated based on the second feature information. The second feature information is the feature information obtained by the instance segmentation network to segment the low-quality image into target instances.

[0061] The processing module is also used to obtain target information based on the basic image features through a detection network. The target information is used to indicate the target object in the image, and the detection network is used to extract the target features of the target object.

[0062] In one possible implementation, the detection network includes at least one of the following: a density map regression network, a point localization network, or an object detection network.

[0063] A fifth aspect of this application provides a data processing apparatus, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any implementation thereof is implemented. For details regarding the steps of the various possible implementations of the first aspect executed by the processor, please refer to the first aspect; further details will not be repeated here.

[0064] A sixth aspect of this application provides a data processing apparatus, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method of either the second aspect or the first aspect described above is implemented. For details regarding the steps of the various possible implementations of the first aspect executed by the processor, please refer to the second aspect; further details will not be repeated here.

[0065] The seventh aspect of this application provides a circuit system including a processing circuit configured to perform a method according to any implementation of the first aspect described above.

[0066] The eighth aspect of this application provides a circuit system including a processing circuit configured to perform the method of any implementation of the second aspect described above.

[0067] The ninth aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any implementation of the first aspect described above.

[0068] The tenth aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any implementation of the second aspect described above.

[0069] The eleventh aspect of this application provides a computer program product that, when run on a computer, causes the computer to execute any implementation of the first aspect described above.

[0070] The twelfth aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform any implementation of the second aspect described above.

[0071] The thirteenth aspect of this application provides a chip device including a processor for calling a program stored in a memory, such that the processor executes the first aspect or any implementation thereof.

[0072] Optionally, the memory may be located inside or outside the chip device.

[0073] The fourteenth aspect of this application provides a chip device including a processor for calling a program stored in a memory, such that the processor executes the second aspect or any implementation thereof described above.

[0074] Optionally, the memory may be located inside or outside the chip device.

[0075] The technical effects of the third aspect or any possible implementation of the third aspect, the fifth aspect, the seventh aspect, the ninth aspect, the eleventh aspect or the thirteenth aspect can be referred to the technical effects of the first aspect or different possible implementations of the first aspect, and will not be repeated here.

[0076] The technical effects of the fourth aspect or any possible implementation of the fourth aspect, the sixth aspect, the eighth aspect, the tenth aspect, the twelfth aspect or the fourteenth aspect can be found in the technical effects of the second aspect or different possible implementations of the second aspect, and will not be repeated here. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 is a schematic diagram of a structural framework for artificial intelligence.

[0079] Figure 2 is a schematic diagram of the functional architecture provided in an embodiment of this application;

[0080] Figure 3 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0081] Figure 4 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0082] Figure 5 is a schematic diagram of the overall architecture of the neural network model provided in the embodiment of this application;

[0083] Figure 6 is a schematic diagram of target information provided in an embodiment of this application;

[0084] Figure 7 is a schematic diagram of NMS dynamic threshold prediction provided in an embodiment of this application;

[0085] Figure 8 is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0086] Figure 9 is a schematic diagram of instance-aware pre-training provided in an embodiment of this application;

[0087] Figure 10 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0088] Figure 11 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0089] Figure 12 is a structural schematic diagram of a terminal device provided in an embodiment of this application;

[0090] Figure 13 is a schematic diagram of a server provided in an embodiment of this application;

[0091] Figure 14 is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0092] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0093] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0094] In the field of computer vision, the main task of crowd counting is to identify and locate individual heads in images, and further assess key information such as crowd density and flow. This technology is particularly useful in public space scenarios, such as parks, shopping malls, or transportation hubs. Traditional methods typically rely on detection models to achieve accurate identification and localization by recognizing feature differences between target objects and the background in an image.

[0095] The applicant's research found that existing detection model architectures have relatively fixed limitations due to their inherent inductive bias problem (the preference selection of different models when processing data). When dealing with complex and varied scenes with dense crowds, especially under the influence of complex factors such as rapid crowd movement and insufficient lighting, the quality of image acquisition is often severely affected. This makes it difficult for models to effectively model human heads with extreme scale variations, resulting in a series of problems such as overestimation / underestimation, missed detections, and false detections.

[0096] To address the aforementioned problems, this application provides a data processing method that can be implemented in conjunction with artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. Using artificial intelligence for data processing is a common application of AI.

[0097] First, the overall workflow of an artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of the main framework of artificial intelligence. The framework is then elaborated on from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0098] (1) Infrastructure

[0099] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0100] (2) Data

[0101] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0102] (3) Data processing

[0103] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0104] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0105] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0106] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0107] (4) General ability

[0108] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0109] (5) Smart Products and Industry Applications

[0110] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0111] The product form of this application embodiment can be an image recognition application. Image recognition applications can run on terminal devices or cloud-based servers.

[0112] The image recognition method described in this application can be applied to various target object counting scenarios. By inputting an image, it determines whether a target object exists in the image and locates its specific position within the image. The target object can be the entirety of an individual or a part of it. For example, in a scenario of estimating the density of crowds in a public space, the "head of an individual in the crowd" can be used as the target object for identification, counting, and location.

[0113] In one possible implementation, image recognition applications can perform high-angle, small-target detection tasks, such as those from the perspective of drones, and obtain processing results.

[0114] In one possible implementation, image recognition applications can also be combined with other task scenarios, such as target detection under infrared light and image models.

[0115] The input image data can be real-time capture from the terminal device's camera or a screenshot taken from an existing video; there are no specific restrictions here.

[0116] In one possible implementation, a user can open an image recognition application installed on a terminal device and input image data. The image recognition application can use a model trained by the method provided in the embodiments of this application or process the image data using the method provided in the embodiments of this application, and present the processing result (marking the target object in the input image data with a bounding box) to the user (the presentation method may include, but is not limited to, displaying, playing, saving, uploading to the cloud, etc.).

[0117] In one possible implementation, a user can open an image recognition application installed on a terminal device and input image data. The image recognition application can then send the image data to a cloud-based server. The cloud-based server processes the image data using a model trained by the method provided in this application embodiment and sends the processing result back to the terminal device. The terminal device can then present the processing result to the user (the presentation method may include, but is not limited to, displaying, playing, saving, or uploading to the cloud).

[0118] The image recognition application in this application will be described below, focusing on its functional architecture and the product architecture that implements its functions.

[0119] Referring to Figure 2, which is a schematic diagram of the functional architecture of an image recognition application in an embodiment of this application:

[0120] In one possible implementation, as shown in FIG2, the image recognition application 102 may receive input parameters 101 (e.g., containing image data) and generate a processing result 103. The image recognition application 102 may execute on at least one computer system (for example) and includes computer code that, when executed by one or more computers, causes the computers to execute a model trained by the methods provided in the embodiments of this application.

[0121] Referring to Figure 3, which illustrates a system architecture, the system may include a terminal 100 and a server 200. The server 200 may include one or more servers (Figure 3 uses one server as an example), and the server 200 may provide video processing or natural language generation functions for one or more terminals.

[0122] The terminal 100 may have an image recognition application installed, or a webpage related to image processing or natural language generation functions opened. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the image processing or natural language generation function interface and send the parameters to the server 200. The server 200 can obtain the image processing result based on the received parameters and return the image processing result to the terminal 100.

[0123] It should be understood that in some optional implementations, the terminal 100 can also complete the task processing action of obtaining the image based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0124] The product form of terminal 100 in Figure 3 is described below;

[0125] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle-mounted device, security device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0126] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of this application will be introduced below.

[0127] (1) Convolutional Neural Network (CNN)

[0128] A CNN (Convolutional Neural Network) is a deep neural network with convolutional structures. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to a subset of neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units within the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0129] (2) Multi-head attention (MHA)

[0130] Multi-head attention (MHA) is an extension of the attention mechanism widely used in deep learning models. In a multi-head attention model, the input features (such as features extracted from a CNN) are first divided into multiple "heads," each of which independently computes attention weights. The input feature sequence is linearly transformed to obtain Query(Q), Key(K), and Value(V) matrices. Each head independently performs Scaled Dot-Product Attention (SMA) to obtain its own attention weights. These weights reflect the importance of different features or regions in the fusion process. Then, each head weights the input features according to its attention weights and generates an output vector. These output vectors are then concatenated to form a unified feature representation.

[0131] Multi-head attention mechanisms can be applied to image feature fusion. Image features include edges, textures, and colors. Different neural networks have different inductive biases for image features, meaning they interpret image features differently. By extracting features from the original image and using a multi-head attention mechanism to focus on multiple important regions or features in the image, a more comprehensive and expressive feature representation can be obtained. Specifically, this fusion process can be achieved through simple addition, weighted fusion, or feature selection-based methods.

[0132] (3) Feature Pyramid Module (FPN)

[0133] FPN generates multi-scale feature representations using feature maps from different levels within a CNN. Specifically, it enhances the spatial resolution of high-level features through upsampling and lateral connections while preserving the semantic information of low-level features. In other words, high-level features (such as deep CNN outputs) have richer semantic information but lower resolution, while low-level features (such as shallow CNN outputs) have higher resolution but less semantic information. FPN generates a feature pyramid encompassing multiple scales by fusing these feature maps from different levels.

[0134] (4) Differentiable binary regression segmentation (DB)

[0135] Binarization (DB) is a method used in deep learning image processing tasks to convert images into black and white pixels with pixel values ​​of 0 and 1. This method introduces a differentiable activation function, allowing the grayscale data of an image to be converted into a binary form, and adaptively learns the optimal threshold during training to optimize the binarization effect.

[0136] (5) Loss Function

[0137] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss function value.

[0138] (6) Backpropagation algorithm

[0139] Neural networks can employ backpropagation (BP) to correct the parameters of the initial prediction model during training, thereby reducing the error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial prediction model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the prediction model, such as the weight matrix.

[0140] (7) Non-maximum suppression (NMS)

[0141] Non-maximum suppression (NMS) suppresses elements that are not local maxima, and can be understood as a search for local maxima. It is commonly used in tasks such as edge detection and object detection to remove redundant and inaccurate results, retaining only the elements most likely to represent true edges or objects. In edge detection, NMS can help refine edges by suppressing all gradient values ​​except for local maxima, retaining only edges with the strongest intensity changes.

[0142] Based on this, an embodiment of this application provides a data processing method. As shown in Figure 4, the data processing method provided in this embodiment includes the following steps 401-403.

[0143] 401. Get Image.

[0144] In one possible implementation, the image is the image from which target object recognition needs to be performed. It can be a picture taken in the detected scene or a frame extracted from a video. The target object to be recognized can be a person, an animal (such as a cat, dog, bird, etc.), a vehicle, etc.; or it can be a part of the target object that is being recognized, such as a person's head or a vehicle's license plate.

[0145] 402. The target feature information of the image is obtained through the target network. The target feature information is the fusion result of the first feature information and the second feature information. The first feature information is obtained by the first sub-network extracting the target features of the image, and the second feature information is obtained by the second sub-network segmenting the target instance of the image. The first sub-network and the second sub-network are two branch networks in the target network.

[0146] Specifically, the first and second sub-networks are two parallel branch networks within the target network (which can also be referred to as the detection model in this application). The first and second sub-networks have different inductive biases for image features; that is, due to differences in structure or training methods, they process image features differently. The first sub-network is responsible for feature extraction from the image, thereby summarizing and identifying the first feature information of the target object in the image. The second sub-network is responsible for instance segmentation of the image, obtaining the second feature information, such as the contour or shape of the target object. Then, by integrating the first and second feature information, feature fusion is performed to obtain target feature information with better performance.

[0147] For example, taking the scenario of estimating crowd density in a spatial setting as an example, the key indicator is the detection of individual heads appearing in the input image. That is, the human head is the target feature that the detection model focuses on.

[0148] Please refer to Figure 5, which shows the overall framework of model reasoning in the embodiments of this application.

[0149] As shown in Figure 5, after the image is input into the detection model, the basic image features are obtained through the feature extraction network (the backbone network of the detection model). Then, these basic image features are passed to two sub-networks: the first sub-network (the detection branch in Figure 5) and the second sub-network (the instance segmentation branch in Figure 5).

[0150] In the feature processing stage, multi-head attention fusion (the upper right part of Figure 5) is used to perform deep processing on the basic image features to obtain a global representation of the image, such as the overall attributes of the image, including contextual information, shape characteristics, and texture details. Next, this global representation is used to guide the feature information extracted from the detection branch and the feature information extracted from the instance segmentation branch to perform feature fusion, resulting in fused feature information.

[0151] It should be understood that the detection branch in this application has two stages: the first stage extracts target features from the input content, and the second stage makes predictions and inferences based on the feature information. The feature fusion step mentioned earlier is embedded between these two stages, providing support for the entire detection process.

[0152] In one possible implementation, the feature extraction network is a neural network consisting of a residual network (ResNet) and an FPN, which can acquire multi-level, multi-scale basic image features.

[0153] For example, the detection network consists of a density map regression network, a point localization network, or an object detection network.

[0154] For example, the instance segmentation network is a differentiable binary regression segmentation network.

[0155] In one possible implementation, after obtaining the global image representation of the input image, the detection features obtained from the object detection branch and the segmentation features obtained from the instance segmentation branch are combined and fused point-by-point using the MHA mechanism. Specifically, the input feature sequence is transformed linearly to obtain Query(Q), Key(K), and Value(V) matrices, and the advantageous features from different branches are extracted and fused. In general, the feature fusion module guided by the global image representation achieves complementary fusion of feature information from the two branches. For each point of the features from the two branches, two vectors can be obtained. By introducing the global representation, dynamic fusion can be achieved among these three through an attention mechanism, resulting in a more comprehensive and expressive feature representation.

[0156] 403. Based on the target feature information, target information is obtained through a prediction network. The target information is used to indicate the target object in the image.

[0157] As shown in Figure 5, the fused target feature information is inferred through the prediction network of the detection branch to obtain target information. In this application, the target information includes the number of target objects and the location of the target objects in the image.

[0158] It is understandable that the target object may not exist in the image, so the target information can also indicate that the target object does not exist in the input image.

[0159] Specifically, the output target information is shown in Figure 6. Detection boxes are marked in the input image, with each detection box representing an individual's head within the crowd. The number of detection boxes indicates the number of target objects (human heads) predicted by the detection model, and the location of each detection box indicates the specific location of the target object (human head).

[0160] Furthermore, this application also proposes a dynamic threshold prediction method for NMS based on feature information obtained from an instance segmentation network. Specifically, when the prediction network in the detection branch performs inference on the image, the CNN divides the image into multiple local regions (patch). As shown in Figure 7, the instance segmentation network obtains a binarized image, where white areas identify the target object (a person's head), while black areas are considered as background without the target object. In general, this process provides an intuitive representation of the distribution of the target object's location in the image. By dividing the image into multiple patches and combining the number (density) of target objects in each patch of the binarized image, the corresponding threshold can be dynamically adjusted for each patch during the prediction process. This includes, but is not limited to, a threshold for verifying the confidence of the detection box, and an overlap threshold required for calculating the intersection over union (IoU) of the detection boxes. As shown in Figure 7, the image is divided into 4x2 patches, and the threshold corresponding to different patches is related to the density of the target object in that patch.

[0161] For example, the IoU threshold is set to 0.5 by default. If the number of target objects in the first patch is greater than the number of target objects in other patches, the IoU threshold in the first patch will be dynamically adjusted to 0.6.

[0162] This application proposes a parallel multi-branch detection model architecture. By leveraging the unique inductive bias between the detection branch and the instance segmentation branch, they work in parallel and complement each other, enhancing the overall performance of the model. Through dynamic fusion of multiple branches based on global representation, the advantageous features of different branches are extracted and fused, improving the expressive power of feature information. Furthermore, the threshold of NMS is dynamically adjusted based on the features of the segmentation branches to predict the detection box results. This method overcomes the limitation of the fixed global threshold in traditional NMS algorithms, making it more flexible in processing detection box results and improving the accuracy and robustness of NMS post-processing.

[0163] Furthermore, the applicant's research found that fine-tuning the backbone network using an instance segmentation network can also improve the target instance perception capability of the detection model. Specifically, instance segmentation is used as a pre-training task to enhance the instance perception capability of the detection model's backbone network. After completing pre-training, the parameters of the backbone network are fine-tuned to adapt to and optimize other related tasks.

[0164] Based on this, the present application embodiment also provides a data processing method, as shown in FIG8. The data processing method provided by the present application embodiment includes the following steps 801-803.

[0165] 801. Get Image.

[0166] Similar to step 401 in Figure 4 above, the specifics will not be repeated here.

[0167] 802. The basic image features of the image are obtained through the feature extraction network. The feature extraction network is a network updated based on the second feature information. The second feature information is the feature information obtained by the instance segmentation network to segment the low-quality image into target instances.

[0168] The training process of the feature extraction network (the backbone network of the detection model) will be described next:

[0169] As shown in Figure 9, during the pre-training stage of the detection model, the instance segmentation network is used to enhance the strength perception capability of the backbone network. The updated weights are then passed to the backbone network during fine-tuning training, so that the updated model can better learn the features of the target object and improve the accuracy and robustness of detection.

[0170] 803. Based on the basic image features, target information is obtained through a detection network. The target information is used to indicate the target object in the image, and the detection network is used to extract the target features of the target object.

[0171] Similarly, during model inference, the image features obtained from the updated feature extraction network based on strength perception are used to extract target features from these image features, ultimately yielding the prediction and inference result, i.e., the target information. The content of the target information is similar to that in the embodiment shown in Figure 4 above, and will not be elaborated further here.

[0172] In this embodiment, the basic image features on which prediction inference is based can be regarded as a comprehensive utilization of the features extracted by the object detection network and the features extracted by the instance segmentation network, which can improve the overall performance and accuracy of the model.

[0173] It's worth noting that both the instance segmentation pre-trained model approach and the multi-branch feature fusion strategy can significantly improve the model's detection performance. Since these two approaches are not mutually exclusive, they can also be combined. Specifically, a parallel multi-branch structure can be incorporated into the instance segmentation pre-trained model, enabling the model to simultaneously process and fuse feature information from multiple branches, thereby further improving the model's performance and accuracy in prediction and inference.

[0174] The following section describes the beneficial effects of the embodiments of this application based on experimental results. The invention is then applied to a real-world security scenario, and the model's performance is tested using a proprietary dataset. The evaluation metrics are normalized absolute error (NAE) and mean absolute error (MAE).

[0175] Where N represents the number of samples tested in the dataset, y i For the model prediction results, The results represent the actual values. The experimental results are shown in Table 1:

[0176] Table 1

[0177] As shown in Table 1, using existing object detection methods, a pedestal model is used to extract general features. After passing through an FPN (Field-Programmable Network), a four-level multi-scale general representation is obtained. The detection result can then be obtained through an object detection head, which learns the positional bias and confidence of each anchor based on preset anchors. Discrete detection boxes are obtained after NMS (Non-Maximum Suppression). It can be seen that the performance at this stage is poor and cannot meet the requirements.

[0178] Next, after using the pre-trained base model parameters of instance segmentation perception, we replaced the base parameters of the previous experiment and retrained the object detection. We can see that the performance has been greatly improved.

[0179] Next, feature extraction for different tasks is performed through parallel multi-task branches. These two branches use an object detection head and a differentiable binary instance segmentation head, respectively, with the instance segmentation head obtaining the binary instance segmentation result. After obtaining the global representation, the features from the two aligned branches are fused point by point, resulting in a further performance improvement.

[0180] Finally, the features from the instance segmentation branches are used through an arbitrary network to obtain features for the NMS dynamic threshold. These features can then regress the NMS threshold for the corresponding patch for NMS post-processing. At this point, optimal results are obtained that far exceed the baseline.

[0181] The methods provided in the embodiments of this application have been described in detail above. Next, the device for performing the above methods provided in the embodiments of this application will be described.

[0182] Please refer to Figure 10, which is a schematic diagram of the structure of a data processing apparatus 1000 provided in an embodiment of this application. As shown in Figure 10, the apparatus includes:

[0183] Acquisition module 1001 is used to acquire images;

[0184] Processing module 1002 is used to obtain target feature information of the image through the target network. The target feature information is the fusion result of first feature information and second feature information. The first feature information is obtained by the first sub-network extracting target features from the image, and the second feature information is obtained by the second sub-network segmenting the image into target instances. The first sub-network and the second sub-network are two branch networks in the target network.

[0185] The processing module 1002 is also used to obtain target information through a prediction network based on target feature information, and the target information is used to indicate the target object in the image.

[0186] In one possible implementation, the processing module 1002 is specifically used for:

[0187] The basic image features of an image are obtained through a feature extraction network, which is the backbone network of the target network.

[0188] The basic image features are input into the first sub-network and the second sub-network respectively to obtain the first feature information and the second feature information;

[0189] Based on the basic image features, the first feature information and the second feature information are fused to obtain the target feature information.

[0190] In one possible implementation, the processing module 1002 is specifically used to: extract a global representation of the image based on the basic image features;

[0191] The target feature information is obtained by fusing the first and second feature information through global representation.

[0192] In one possible implementation, before obtaining the basic image features of the image through the feature extraction network, the processing module 1002 is further configured to:

[0193] The feature extraction network is updated using the second feature information.

[0194] In one possible implementation, before obtaining the target information through a prediction network based on the target feature information, the processing module 1002 is further configured to:

[0195] Based on the second feature information, a first position distribution is obtained, which is used to indicate the position information of at least one target object in the image;

[0196] Based on the first position distribution, multiple threshold values ​​are obtained; wherein, the first threshold value is used to indicate the confidence threshold or overlap threshold of the output result of the prediction model in the first image region, the confidence threshold or overlap threshold is related to the density of the target object in the first image region, the first prediction information is one of the multiple prediction information, and the first image region is a region in the image;

[0197] The processing module 1002 is specifically used to: obtain target information through a prediction network based on target feature information and multiple threshold values.

[0198] In one possible implementation, the first sub-network includes at least one of the following: a density map regression network, a point localization network, or an object detection network.

[0199] Please refer to Figure 11, which is a schematic diagram of the structure of a data processing apparatus 1100 provided in an embodiment of this application. As shown in Figure 11, the apparatus includes:

[0200] Acquisition module 1101 is used to acquire images;

[0201] Processing module 1102 is used to obtain basic image features of the image through a feature extraction network. The feature extraction network is a network updated based on second feature information. The second feature information is feature information obtained by the instance segmentation network to segment the low-quality image into target instances.

[0202] The processing module 1102 is also used to obtain target information based on the basic image features through a detection network. The target information is used to indicate the target object in the image, and the detection network is used to extract the target features of the target object.

[0203] In one possible implementation, the detection network includes at least one of the following: a density map regression network, a point localization network, or an object detection network.

[0204] This application also relates to a terminal device. Figure 12 is a structural schematic diagram of a terminal device provided in this application embodiment. As shown in Figure 12, the terminal device 1200 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited here. The terminal device 1200 may be equipped with the data processing device described in the embodiment corresponding to Figure 10 or Figure 11. Specifically, the terminal device 1200 includes: a receiver 1210, a transmitter 1220, a processor 1230, and a memory 1240 (the number of processors 1230 in the terminal device 1200 can be one or more; Figure 12 shows one processor as an example). The processor 1230 may include an application processor 1231 and a communication processor 1232. In some embodiments of this application, the receiver 1210, transmitter 1220, processor 1230, and memory 1240 can be connected via a bus or other means.

[0205] Memory 1240 may include read-only memory and random access memory, and provides instructions and data to processor 1230. A portion of memory 1240 may also include non-volatile random access memory (NVRAM). Memory 1240 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0206] Processor 1230 controls the operation of the terminal device. In specific applications, the various components of the terminal device are coupled together through a bus system. This bus system includes not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0207] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1230. The processor 1230 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1230 or by instructions in software form. The processor 1230 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1230 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1240. Processor 1230 reads information from memory 1240 and, in conjunction with its hardware, completes the steps of the above method.

[0208] Receiver 1210 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the terminal device. Transmitter 1220 can be used to output digital or character information through the first interface; transmitter 1220 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1220 may also include a display device such as a display screen.

[0209] This application also provides a server. Referring to Figure 13, which is a schematic diagram of a server structure provided in this application embodiment, the server 1300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1314 (e.g., one or more processors) and memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) for storing application programs 1342 or data 1344. The memory 1332 and storage media 1330 can be temporary or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server. Furthermore, the CPU 1314 may be configured to communicate with the storage media 1330 and execute the series of instruction operations in the storage media 1330 on the server 1300.

[0210] Server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358; or, one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0211] In this embodiment, the central processing unit 1314 is used to perform actions related to model training or model inference in the above embodiments.

[0212] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned terminal device, or causes the computer to perform steps as performed by the aforementioned training device.

[0213] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned terminal device, or causes the computer to perform steps as performed by the aforementioned training device.

[0214] The terminal device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the terminal device to execute the data processing method described in the above embodiments, or to cause the chip in the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0215] Specifically, please refer to Figure 14, which is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1400. The NPU 1400 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1403, which is controlled by the controller 1404 to extract matrix data from the memory and perform multiplication operations.

[0216] In some implementations, the arithmetic circuit 1403 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1403 is a two-dimensional pulsating array. The arithmetic circuit 1403 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1403 is a general-purpose matrix processor.

[0217] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1402 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1401 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1408.

[0218] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 via Direct Memory Access Controller (DMAC) 1405. Input data is also transferred to unified memory 1406 via DMAC.

[0219] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1409.

[0220] The Bus Interface Unit (BIU) 1413 is used by the instruction fetch memory 1409 to fetch instructions from external memory, and also by the memory access controller 1405 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0221] The DMAC is mainly used to move input data from external memory DDR to unified memory 1406, or to weight data to weight memory 1402, or to input data to input memory 1401.

[0222] The vector computation unit 1407 includes multiple processing units that further process the output of the computation circuit 1403 when necessary, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of the predicted label plane.

[0223] In some implementations, the vector computation unit 1407 can store the processed output vector in the unified memory 1406. For example, the vector computation unit 1407 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1403, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1407 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1403, for example, for use in subsequent layers of the neural network.

[0224] The instruction fetch buffer 1409 connected to the controller 1404 is used to store the instructions used by the controller 1404;

[0225] Unified memory 1406, input memory 1401, weighted memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0226] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0227] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0228] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0229] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0230] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized by, The method comprises the following steps: acquiring an image; obtaining target feature information of the image through a target network, the target feature information being a fusion result of first feature information and second feature information, the first feature information being obtained by target feature extraction on the image through a first sub-network, the second feature information being obtained by target instance segmentation on the image through a second sub-network, the first sub-network and the second sub-network being two branch networks in the target network; obtaining target information through a prediction network according to the target feature information, the target information being used for indicating a target object in the image.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining basic image features of the image through a feature extraction network, the feature extraction network being a backbone network of the target network; inputting the basic image features into the first sub-network and the second sub-network respectively to obtain the first feature information and the second feature information; performing feature fusion on the first feature information and the second feature information based on the basic image features to obtain the target feature information.

3. The method of claim 2, wherein, The method comprises the following steps: extracting global representation of the image according to the basic image features; performing feature fusion on the first feature information and the second feature information through the global representation to obtain target feature information.

4. The method according to claim 2 or 3, characterized in that, Before the step of obtaining the basic image features of the image through the feature extraction network, the method further comprises the following step: updating the feature extraction network through the second feature information.

5. The method according to any one of claims 1-4, characterized in that, Before the step of obtaining target information through the prediction network according to the target feature information, the method further comprises the following steps: obtaining a first position distribution according to the second feature information, the first position distribution being used for indicating position information of at least one target object in the image; obtaining a plurality of threshold values according to the first position distribution; wherein a first threshold value is used for indicating a confidence threshold value or an overlapping degree threshold value of an output result of the prediction model in a first image region, the confidence threshold value or the overlapping degree threshold value being related to a density degree of the target object in the first image region, the first prediction information being one of the plurality of prediction information, and the first image region being one region in the image; The method comprises the following steps: obtaining target information through the prediction network according to the target feature information and the plurality of threshold values.

6. The method according to any one of claims 1-5, characterized in that, The first sub-network comprises at least one of the following: a density map regression network, a point positioning network or a target detection network.

7. A data processing method, characterized by, The method comprises the following steps: acquiring an image; obtaining basic image features of the image through a feature extraction network, the feature extraction network being a network updated based on second feature information, the second feature information being feature information obtained by target instance segmentation on a low-quality image through an instance segmentation network; According to the basic image feature, target information is obtained through a detection network, the target information being used to indicate a target object in the image, and the detection network being used to extract a target feature of the target object.

8. The method of claim 7, wherein, The detection network comprises at least one of a density map regression network, a point positioning network or a target detection network.

9. A data processing apparatus, characterized by, Comprise: An acquisition module is configured to acquire an image; A processing module is configured to obtain target feature information of the image through a target network, the target feature information being a fusion result of first feature information and second feature information, the first feature information being obtained by a first sub-network performing target feature extraction on the image, and the second feature information being obtained by a second sub-network performing target instance segmentation on the image, the first sub-network and the second sub-network being two branch networks in the target network; The processing module is further configured to obtain target information through a prediction network according to the target feature information, the target information being used to indicate a target object in the image.

10. The apparatus of claim 9, wherein, The processing module is specifically configured to: obtain basic image features of the image through a feature extraction network, the feature extraction network being a backbone network of the target network; input the basic image features into the first sub-network and the second sub-network respectively to obtain the first feature information and the second feature information; perform feature fusion on the first feature information and the second feature information based on the basic image features to obtain the target feature information.

11. The apparatus of claim 10, wherein, The processing module is specifically configured to: obtain global representation of the image according to the basic image features; perform feature fusion on the first feature information and the second feature information based on the global representation to obtain target feature information.

12. The apparatus of claim 10 or 11, wherein, Before the basic image features of the image are obtained through the feature extraction network, the processing module is further configured to: update the feature extraction network through the second feature information.

13. The apparatus of any one of claims 9-12, wherein, Before the target information is obtained through the prediction network according to the target feature information, the processing module is further configured to: obtain a first position distribution according to the second feature information, the first position distribution being used to indicate position information of at least one target object in the image; obtain a plurality of threshold values according to the first position distribution; wherein a first threshold value is used to indicate a confidence threshold or an overlap threshold of an output result of the prediction model in a first image region, the confidence threshold or the overlap threshold being related to a density degree of the target object in the first image region, the first prediction information being one of the plurality of prediction information, and the first image region being one region in the image; The processing module is specifically configured to: obtain the target information through the prediction network according to the target feature information and the plurality of threshold values.

14. The apparatus of any one of claims 9-13, wherein, The first sub-network comprises at least one of a density map regression network, a point positioning network or a target detection network.

15. A data processing apparatus, characterized by Comprise: An acquisition module is configured to acquire an image; The processing module is configured to obtain a basic image feature of the image by using a feature extraction network, the feature extraction network being a network updated based on second feature information, the second feature information being feature information obtained by performing target instance segmentation on the low-quality image by using an instance segmentation network. The processing module is further configured to obtain target information by using a detection network according to the basic image feature, the target information being used to indicate a target object in the image, and the detection network being used to extract a target feature of the target object.

16. The apparatus of claim 15, wherein, The detection network includes at least one of a density map regression network, a point positioning network, or a target detection network.

17. A computer storage medium, comprising, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 6, or cause the one or more computers to implement the operations of the method of claim 7 or 8.

18. A computer program product, characterised in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 6, or cause the one or more computers to implement the operations of the method of claim 7 or 8.

19. A system comprising at least one processor, at least one memory; the processor, the memory are connected through a communication bus and complete the communication between each other; The at least one memory is configured to store code; The at least one processor is configured to execute the code to perform the method of any one of claims 1 to 6, or to perform the method of claim 7 or 8.

20. A chip comprising a processor, wherein: The processor is configured to support a data processing apparatus to implement the method of any one of claims 1 to 6, or to implement the method of claim 7 or 8.

Images