A model training method and device thereof

Abstract

The application discloses a model training method, which comprises the following steps: sampling a sample image to obtain a first image block and a second image block; performing feature extraction on the first image block and the second image block by a feature extraction network based on the fact that the intersection-over-union ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold; determining a loss according to the difference between the obtained first feature map and second feature map; and updating the feature extraction network based on the loss. The application uses the intersection-over-union ratio as a way to measure the relative distance between two image blocks, ensures the semantic consistency in a local region to restore the basic assumption of contrast learning by requiring the intersection-over-union ratio of the image blocks to be greater than a given threshold, limits the image blocks within a local region, thereby converts global inconsistency into local consistency, and further improves the data processing accuracy of the updated feature extraction network.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model training method and apparatus thereof. Background Technology

[0002] Computer vision is an integral part of various intelligent / autonomous systems across diverse application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and the military. It's the science of how to use cameras / video cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's about equipping computers with eyes (cameras or video cameras) and a brain (algorithms) to replace human eyes in identifying, tracking, and measuring targets, thus enabling computers to perceive their environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be viewed as the science of how to enable artificial systems to "perceive" from images or multidimensional data. In short, computer vision uses various imaging systems to replace visual organs in acquiring input information, and then the computer replaces the brain in processing and interpreting this input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, possessing the ability to autonomously adapt to their environment.

[0003] Perceptual networks are neural network models that process and analyze images to obtain results. Currently, perceptual networks can perform an increasing number of functions, such as image classification, 2D detection, semantic segmentation, key point detection, linear object detection (e.g., lane or stop line detection in autonomous driving technology), and drivable area detection. Furthermore, visual perception systems are characterized by low cost, non-contact operation, small size, and large information capacity. With the continuous improvement in the accuracy of visual perception algorithms, they have become a key technology in many artificial intelligence systems and are being used more and more widely. Examples include: the identification of dynamic obstacles (people or vehicles) and static objects (traffic lights, traffic signs, or traffic cones) on the road surface in advanced driving assistance systems (ADAS) and autonomous driving systems (ADS); and the slimming effect achieved in the beautification function of terminal vision photos by recognizing human masks and key points.

[0004] In recent years, self-supervised representation learning based on contrastive learning has achieved significant technological breakthroughs. Its basic assumption is that image patches from the same image describe the same semantic information, and their features should be as similar as possible. Conversely, image patches from different images describe different semantic information, and therefore their features should be as dissimilar as possible. While this method does not require manual annotation, it still requires the additional single-instance assumption, which requires that each image contains only one instance, and this instance occupies the central and main part of the image. This is necessary to consider image patches from the same image as positive samples and image patches from different images as negative samples. However, in autonomous driving scenarios, street scene images often contain multiple instances such as pedestrians and vehicles, exhibiting strong global inconsistency and not conforming to the single-instance assumption. Therefore, existing methods are often limited to datasets with the single-instance assumption and struggle to fully utilize more realistic multi-instance autonomous driving datasets. Summary of the Invention

[0005] Firstly, this application provides a model training method, the method comprising:

[0006] Obtain sample images; the sample images can be multi-instance images (e.g., street view images) in the above-mentioned autonomous driving scenario. In one possible implementation, the sample images may include multiple objects, including at least one of people, vehicles, traffic signs, lane lines, and plants. For example, objects may include, but are not limited to, at least one of the following: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, bicycles), traffic signs (TrafficSign, GuideSign, Billboard, red traffic lights / yellow traffic lights / green traffic lights / black traffic lights, RoadSign).

[0007] The sample image is sampled to obtain a first image block and a second image block, wherein the first image block and the second image block are different image blocks on the sample image;

[0008] After obtaining the sample image, the sample image can be sampled to obtain multiple local images (e.g., a first image block and a second image block). The first image block and the second image block can be small images obtained by sampling the sample image (small means that the image area of ​​the first image block and the second image block is smaller relative to the size of the sample image).

[0009] The above sampling can be understood as cropping. In one sampling process, a rectangular region can be randomly determined on the sample image, and the image within the determined rectangular region is used as the first image block. In another sampling process, a rectangular region can be randomly determined on the sample image, and the image within the determined rectangular region is used as the second image block.

[0010] The above sampling can be random sampling. Random sampling can be understood as the sampling location and / or the sampling size being randomly determined.

[0011] The first image block and the second image block can be rectangular images.

[0012] Based on the fact that the intersection ratio between the regions where the first image block and the second image block are located in the sample image is greater than a threshold, feature extraction networks are used to extract features from the first image block and the second image block respectively to obtain a first feature map and a second feature map.

[0013] There may be a common area (overlapping area) between the area where the first image block is located in the sample image and the area where the second image block is located in the sample image. The ratio of the area of ​​the common area to the total area occupied by the first image block and the second image block in the sample image is the intersection over union (IoU) between the areas where the first image block and the second image block are located in the sample image.

[0014] In one possible implementation, the threshold can be a value greater than or equal to 0.4, for example, the threshold can be 0.4, 0.45, 0.5, 0.55, etc. It should be understood that the intersection-union ratio (IURR) threshold obviously cannot be set too low, but at the same time, it is not better to be as high as possible, because it is undesirable for image patches to be completely unrelated, but it is also undesirable for them to be completely identical. Therefore, the choice of the IURR threshold is actually controlling the balance between data noise and data complexity in multi-instance self-supervised learning.

[0015] Based on the difference between the first feature map and the second feature map, a loss is determined, and the feature extraction network is updated based on the loss to obtain an updated feature extraction network.

[0016] This application uses the intersection-over-union ratio (IoU) as a measure of the relative distance between two image patches. By requiring the IoU of image patches to be greater than a given threshold, it ensures semantic consistency within local regions to restore the basic assumptions of contrastive learning. This restricts image patches to a local region, thereby transforming global inconsistency into local consistency and improving the data processing accuracy of the updated feature extraction network. Simultaneously, the choice of threshold can effectively control the balance between data noise and data complexity in different scenarios.

[0017] In one possible implementation, the threshold is a value greater than or equal to 0.4.

[0018] In one possible implementation, before determining the loss based on the difference between the first feature map and the second feature map, the method further includes: aligning the first image block and the second image block to obtain an aligned first image block and an aligned second image block; determining the loss based on the difference between the first feature map and the second feature map includes: determining the loss based on the difference between the aligned first feature map and the aligned second feature map.

[0019] In one possible implementation, the sample image includes a target region, which is the overlapping area where the first image block and the second image block are located on the sample image. Aligning the first image block and the second image block includes: determining, based on the target region, a first sub-feature map in the first feature map corresponding to the target region and a second sub-feature map in the second feature map corresponding to the target region; upsampling the first sub-feature map to obtain the aligned first image block; and upsampling the second sub-feature map to obtain the aligned second image block, wherein the aligned first image block and the aligned second image block have the same size.

[0020] To distinguish features across multiple instances, the global pooling layer following the backbone network needs to be discarded to preserve the two-dimensional structure and positional information of the feature map. However, this also introduces the additional problem of feature misalignment, meaning that the two-dimensional feature map no longer has a one-to-one correspondence of the same relative positions. This application provides two different methods for feature alignment: Region of Interest (ROI) alignment, which treats the overlapping parts of image patches as ROIs for the two image patches respectively. For example, RoI Align can be used to extract only the features of the overlapping parts for subsequent calculations. This method is intuitive but does not fully utilize the information of the non-overlapping parts.

[0021] In one possible implementation, the first feature map and the second feature map are of the same size. The first feature map includes M first feature points, and the second feature map includes M second feature points. The M first feature points correspond to M first pixels in the sample image, and the M second feature points correspond to M second pixels in the sample image. The M first pixels and the M second pixels are in one-to-one correspondence. The method further includes: obtaining a third feature map based on the M first pixels and the M second pixels. The third feature map is of the same size as the second feature map. The third feature map includes M third feature points, each of which is obtained based on the pixel position difference between the corresponding first pixels and second pixels. The third feature map is then fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block.

[0022] In one possible implementation, fusing the third feature map with the first feature map includes:

[0023] The third feature map is stitched together with the first feature map in the depth direction.

[0024] In this process, displacement alignment takes each pair of pixels located at the same relative position, calculates their coordinate displacement in the original image, and concatenates them with the feature map (by splicing them in the depth direction). This is provided as additional lateral information to the prediction network for implicit feature alignment to help subsequent feature prediction. This process makes full use of the feature information of non-overlapping regions and makes the subsequent similarity measurement more flexible.

[0025] In one possible implementation, determining the loss based on the difference between the first feature map and the second feature map includes: processing M first feature points of the first feature map using a target prediction network to obtain a predicted value for each first feature point; clustering the M second feature points of the first feature map based on a target clustering algorithm to update the feature values ​​of the M second feature points, wherein the feature value of each updated second feature point is the cluster center feature value of its cluster category; and determining the loss based on the difference between the predicted value of each first feature point and the feature value of each updated second feature point.

[0026] In this embodiment, image clustering is used, which enables the network to distinguish features of different instances. On the other hand, by considering the overall information of the two-dimensional feature map, the regression target provided by the target branch is more robust, while the online branch introduces a global perspective by deploying a self-attention module to obtain more accurate prediction results.

[0027] In one possible implementation, the method further includes:

[0028] The target prediction network is updated based on the loss.

[0029] In one possible implementation, the sample image includes multiple objects, including at least one of people, vehicles, traffic signs, lane lines, and plants.

[0030] In one possible implementation, the method further includes:

[0031] The target network and the image to be processed are obtained. The target network includes the updated feature extraction network and the downstream task network. The image to be processed is processed through the target network to obtain the processing result.

[0032] Secondly, this application provides a model training apparatus, the apparatus comprising:

[0033] The acquisition module is used to acquire sample images;

[0034] A sampling module is used to sample the sample image to obtain a first image block and a second image block, wherein the first image block and the second image block are different image blocks on the sample image;

[0035] The feature extraction module is used to extract features from the first image block and the second image block respectively through a feature extraction network based on the fact that the intersection ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold, so as to obtain a first feature map and a second feature map.

[0036] The model update module is used to determine the loss based on the difference between the first feature map and the second feature map, and update the feature extraction network based on the loss to obtain the updated feature extraction network.

[0037] In one possible implementation, the threshold is a value greater than or equal to 0.4.

[0038] In one possible implementation, the device further includes:

[0039] An alignment module is used to align the first image block and the second image block before determining the loss based on the difference between the first feature map and the second feature map, so as to obtain an aligned first image block and an aligned second image block.

[0040] The model update module is specifically used for:

[0041] The loss is determined based on the difference between the aligned first feature map and the aligned second feature map.

[0042] In one possible implementation, the sample image includes a target region, which is the overlapping area where the first image block and the second image block are located on the sample image. The alignment module is specifically used for:

[0043] Based on the target region, a first sub-feature map corresponding to the target region in the first feature map and a second sub-feature map corresponding to the target region in the second feature map are determined.

[0044] The first sub-feature map is upsampled to obtain the aligned first image block;

[0045] The second sub-feature map is upsampled to obtain the aligned second image block, wherein the aligned first image block has the same size as the aligned second image block.

[0046] In one possible implementation, the first feature map and the second feature map have the same size. The first feature map includes M first feature points, and the second feature map includes M second feature points. The M first feature points correspond to M first pixels in the sample image, and the M second feature points correspond to M second pixels in the sample image. There is a one-to-one correspondence between the M first pixels and the M second pixels. The alignment module is specifically used for:

[0047] A third feature map is obtained based on the M first pixels and the M second pixels. The third feature map and the second feature map have the same size. The third feature map includes M third feature points, each of which is obtained based on the pixel position difference between the corresponding first pixels and second pixels.

[0048] The third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block.

[0049] In one possible implementation, the alignment module is specifically used for:

[0050] The third feature map is stitched together with the first feature map in the depth direction.

[0051] In one possible implementation, the model update module is specifically used for:

[0052] The target prediction network processes the M first feature points of the first feature map to obtain the predicted value of each first feature point.

[0053] Based on the target clustering algorithm, the M second feature points of the first feature map are clustered to update the feature values ​​of the M second feature points, wherein the feature value of each updated second feature point is the cluster center feature value of the cluster category in which it belongs.

[0054] The loss is determined based on the difference between the predicted value of each first feature point and the feature value of each updated second feature point.

[0055] In one possible implementation, the model update module is further configured to:

[0056] The target prediction network is updated based on the loss.

[0057] In one possible implementation, the sample image includes multiple objects, including at least one of people, vehicles, traffic signs, lane lines, and plants.

[0058] In one possible implementation, the device further includes:

[0059] The data processing module is used to acquire the target network and the image to be processed, wherein the target network includes the updated feature extraction network and the downstream task network;

[0060] The target network is used to process the image to be processed in order to obtain the processing result.

[0061] Thirdly, embodiments of this application provide a model training apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect and any optional method of the first aspect.

[0062] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional method of the first aspect.

[0063] Fifthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the first aspect and any of its optional methods described above.

[0064] Sixthly, this application provides a chip system including a processor for supporting an execution device or training device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices.

[0065] This application provides a model training method, comprising: acquiring a sample image; sampling the sample image to obtain a first image patch and a second image patch, wherein the first image patch and the second image patch are different image patches on the sample image; based on the intersection-over-union ratio (IoU) between the regions where the first image patch and the second image patch are located on the sample image being greater than a threshold, performing feature extraction on the first image patch and the second image patch respectively through a feature extraction network to obtain a first feature map and a second feature map; determining a loss based on the difference between the first feature map and the second feature map, and updating the feature extraction network based on the loss to obtain an updated feature extraction network. Using IoU as a measure of the relative distance between two image patches, by requiring the IoU of image patches to be greater than a given threshold to ensure semantic consistency within a local region to restore the basic assumption of contrastive learning, the image patch is restricted to a local region, thereby transforming global inconsistency into local consistency, and thus improving the data processing accuracy of the updated feature extraction network. Simultaneously, the selection of the threshold can effectively control the balance between data noise and data complexity in different scenarios. Attached Figure Description

[0066] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0067] Figure 2 This is an application scenario for an embodiment of this application;

[0068] Figure 3 This is a schematic diagram of the convolutional neural network structure used in the embodiments of this application;

[0069] Figure 4 This is a schematic diagram of the convolutional neural network structure used in the embodiments of this application;

[0070] Figure 5 A schematic diagram of a system architecture provided for an embodiment of this application;

[0071] Figure 6 This is a schematic diagram of a model training method provided in an embodiment of this application;

[0072] Figure 7 This application provides a schematic diagram of crossover ratio calculation.

[0073] Figure 8 This application provides a schematic diagram of image block sampling as an embodiment.

[0074] Figure 9 This application provides a schematic diagram of image block sampling as an embodiment.

[0075] Figure 10This application provides a schematic diagram of image block sampling as an embodiment.

[0076] Figure 11 This application provides a schematic diagram of image block sampling as an embodiment.

[0077] Figure 12 This is a schematic diagram of a backbone network structure;

[0078] Figure 13 This is a schematic diagram of image block alignment according to an embodiment of this application;

[0079] Figure 14 This is a schematic diagram of image block alignment according to an embodiment of this application;

[0080] Figure 15 This is a clustering illustration of an embodiment of this application;

[0081] Figure 16 This is a schematic diagram of similarity calculation according to an embodiment of this application;

[0082] Figure 17 This is a schematic diagram of a model training method provided in an embodiment of this application;

[0083] Figure 18 This is a schematic diagram of a model training method provided in an embodiment of this application;

[0084] Figure 19 This is a schematic diagram of a downstream task network structure;

[0085] Figure 20 This is a schematic diagram of a head structure;

[0086] Figure 21 This is an illustration of a clustering result;

[0087] Figure 22 A schematic diagram of a model training device provided in an embodiment of this application;

[0088] Figure 23 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0089] Figure 24 This is a schematic diagram of a training device provided in an embodiment of this application;

[0090] Figure 25 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0091] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0092] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0093] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0094] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0095] (1) Infrastructure

[0096] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0097] (2) Data

[0098] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0099] (3) Data processing

[0100] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0101] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0102] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0103] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0104] (4) General ability

[0105] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0106] (5) Smart Products and Industry Applications

[0107] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0108] The following sections provide a brief introduction to application scenarios such as ADAS / ADS visual perception systems, mobile phone beautification, image classification, and product classification.

[0109] Application Scenario 1: ADAS / ADS Visual Perception System

[0110] like Figure 2As shown, ADAS and ADS require real-time detection of multiple types of 2D objects, including: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, bicycles), and traffic signs (traffic signs, guide signs, billboards, red / yellow / green / black traffic lights, road signs). Furthermore, to accurately determine the area occupied by dynamic obstacles in 3D space, 3D estimation of the dynamic obstacles is also required, outputting 3D bounding boxes. To fuse data with LiDAR, it is necessary to acquire the mask of dynamic obstacles, thereby filtering out the laser point clouds hitting the dynamic obstacles; for accurate parking, it is necessary to simultaneously detect the four key points of the parking space; for mapping and localization, it is necessary to detect the key points of static targets. The model trained using the technical solution provided in the embodiments of this application (e.g., the target network) can perform all or part of the above functions in the target network.

[0111] Application Scenario 2: Mobile Phone Beauty Function

[0112] In a mobile phone, a model (e.g., a target network) trained using the technical solution provided in this application can detect the mask and key points of the human body, and can enlarge or reduce the corresponding parts of the human body, such as performing waist slimming and buttock shaping operations, thereby outputting a beautified image.

[0113] Application Scenario 3: Image Classification Scenario:

[0114] After acquiring the image to be classified, a model (e.g., a target network) trained using the technical solution provided in this application can determine the category of objects in the image. Then, the image can be classified according to the category of objects in the image. Photographers take many photos every day, including animals, people, and plants. The method of this application can quickly classify photos according to their content, such as photos containing animals, photos containing people, and photos containing plants.

[0115] Application Scenario 4: Product Classification

[0116] After acquiring an image containing goods, a model (e.g., a target network) trained using the technical solution provided in this application can determine the category of the goods in the image, and then classify the goods according to the category. This is suitable for the wide variety of goods in large shopping malls or supermarkets.

[0117] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0118] (1) Neural Network

[0119] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0120]

[0121] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0122] (2) Object detection: Using image processing and machine learning, computer graphics and other related methods, object detection can determine the category of objects in an image and determine the detection box used to locate the objects.

[0123] (3) A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer refers to the layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn reasonable weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0124] CNN is a very common type of neural network. Below, we will combine... Figure 3 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the input image.

[0125] like Figure 3 As shown, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional / pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230.

[0126] Convolutional / pooling layers 220:

[0127] Convolutional layers:

[0128] like Figure 3The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0129] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0130] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size also have the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0131] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0132] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0133] Pooling layer:

[0134] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 3 Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0135] Fully connected layer 230:

[0136] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize fully connected layers 230 to generate one or a set of outputs representing the required number of classes. Therefore, the fully connected layers 230 can include multiple hidden layers (such as...). Figure 3 As shown in 231, 232 to 23n), the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0137] After the multiple hidden layers in the fully connected layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has propagated forward (e.g., ... Figure 3 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 3 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0138] It should be noted that, as Figure 3 The convolutional neural network 200 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, for example, including only... Figure 3 As shown in the network structure, for example, the convolutional neural network used in the embodiments of this application may only include an input layer 210, a convolutional / pooling layer 220, and an output layer 240.

[0139] It should be noted that, as Figure 3 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 4 The multiple convolutional / pooling layers shown are implemented in parallel, and the extracted features are all input into the fully connected layer 230 for processing.

[0140] (4) Deep Neural Networks

[0141] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0142] (5) Loss Function

[0143] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0144] (6) Backpropagation algorithm

[0145] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0146] (7) Self-supervised learning: This refers to using certain attributes of the data itself (such as rotation angle and image patch distribution order) as "labels" for unsupervised pre-training in a supervised manner. Finally, we take the backbone network parameters as the initialization of network parameters for various downstream tasks. Here, upstream refers to the pre-training process, while downstream refers to various practical vision problems. Among them, contrastive learning is a recent research hotspot. It constructs contrast tasks by mining the consistency of the data itself, assuming that different image patches from the same image should have similar features. The feature representations learned in this way have even achieved transfer effects that surpass supervised pre-training.

[0147] (7) Domain-specific pre-training: This refers to pre-training on an upstream dataset within the same data domain for a downstream task in a specific domain, thereby reducing the data differences between upstream and downstream tasks and lowering the optimization difficulty during downstream fine-tuning. In this invention, both upstream and downstream datasets are multi-instance autonomous driving datasets.

[0148] (9) Backbone: This is the basic network structure for feature extraction from data and the main learning object during pre-training. Based on the backbone, we can deploy different dedicated network structures according to different tasks; the backbone is shareable and transferable, but dedicated networks are often not transferable. For images, the commonly used backbone is a convolutional neural network, whose final feature representation is a two-dimensional feature map (2D feature map). Existing contrastive learning models often deploy an additional global pooling layer, such as averaging in the spatial dimension, processing the two-dimensional feature map into a one-dimensional feature vector (1D feature vector). In this invention, the global pooling layer is discarded, and the two-dimensional feature map is modeled directly.

[0149] The following section describes the more detailed architecture of the execution entity of the model training method in the embodiments of this application.

[0150] The following is combined Figure 5 The system architecture provided in the embodiments of this application will be described in detail. Figure 5 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 5 As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0151] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0152] The data acquisition device 560 is used to collect training samples. Training samples can be image data, etc.; in this embodiment, training samples can be sample images. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0153] Training device 520 can train the feature extraction network based on training samples maintained in database 530 to obtain target model / rule 501. In this embodiment, target model / rule 501 can be the updated feature extraction network. It should be understood that the above-described training process of the feature extraction network can be a pre-training process. After obtaining the updated feature extraction network, it can be fine-tuned for the target task by combining it with downstream task datasets.

[0154] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0155] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 5 The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud, etc.

[0156] Specifically, the training device 520 can transmit the data to the execution device 510.

[0157] exist Figure 5 In the process, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. Users can input data (such as the data to be processed in the embodiments of this application) into the I / O interface 512 through the client device 540.

[0158] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0159] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0160] Finally, the I / O interface 512 presents the processing result (e.g., the processing result in the embodiments of this application) to the client device 540, thereby providing it to the user.

[0161] exist Figure 5 In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0162] It is worth noting that, Figure 5This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 5 In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0163] From the inference side of the model:

[0164] In this embodiment, the computing module 511 of the execution device 520 can obtain the code stored in the data storage system 550 to implement the model feedforward process in this embodiment.

[0165] In this embodiment of the application, the computing module 511 of the execution device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0166] Specifically, the computing module 511 of the execution device 520 can be a hardware system with the function of executing instructions. The data processing method provided in this application embodiment can be software code stored in the memory. The computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to realize the model feedforward process in this application embodiment.

[0167] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps of the data processing method provided in the embodiments of this application can also be implemented by the hardware system without the function of executing instructions in the computing module 511 of the execution device 520, which is not limited here.

[0168] From the training side of the model:

[0169] In this embodiment of the application, the training device 520 can access the memory ( Figure 5 The model training method in this application embodiment can be implemented by code stored in the training device 520 (not shown in the figure) or separately deployed from the training device 520.

[0170] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0171] Specifically, the training device 520 can be a hardware system with instruction execution capabilities. The data processing method provided in this application embodiment can be software code stored in a memory. The training device 520 can retrieve the software code from the memory and execute the retrieved software code to implement the model training method provided in this application embodiment.

[0172] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps of the model training method provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0173] Reference Figure 6 , Figure 6 This is an illustrative embodiment of a model training method provided in this application. The model training method provided in this application can be applied to a training device, which can be a terminal device such as a mobile phone, tablet, laptop, or smart wearable device. The training device can also be a server, chip, or other device with data processing capabilities. Figure 6 As shown, a model training method provided in this application embodiment may include:

[0174] 601. Obtain sample images.

[0175] In autonomous driving scenarios, street view images often contain multiple instances (or objects) such as pedestrians and vehicles, exhibiting strong global inconsistency. This does not conform to the single instance assumption (image patches from the same image describe the same semantic information, and their features should be as similar as possible, while image patches from different images describe different semantic information, and therefore their features should be as dissimilar as possible). As a result, existing methods are often limited to datasets with the single instance assumption, making it difficult to fully utilize more realistic multi-instance autonomous driving datasets.

[0176] In this embodiment of the application, the sample image can be a multi-instance image (e.g., a street view image) under the above-mentioned autonomous driving scenario.

[0177] In one possible implementation, the sample image may include multiple objects, including at least one of people, vehicles, traffic signs, lane lines, and plants. For example, objects may include, but are not limited to, at least one of the following: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, bicycles), traffic signs (TrafficSign, GuideSign, billboard, red traffic lights / yellow traffic lights / green traffic lights / black traffic lights, road signs).

[0178] 602. Sample the sample image to obtain a first image block and a second image block, wherein the first image block and the second image block are different image blocks on the sample image.

[0179] After obtaining the sample image, the sample image can be sampled to obtain multiple local images (e.g., a first image block and a second image block). The first image block and the second image block can be small images obtained by sampling the sample image (small means that the image area of ​​the first image block and the second image block is smaller relative to the size of the sample image).

[0180] The above sampling can be understood as cropping. In one sampling process, a rectangular region can be randomly determined on the sample image, and the image within the determined rectangular region is used as the first image block. In another sampling process, a rectangular region can be randomly determined on the sample image, and the image within the determined rectangular region is used as the second image block.

[0181] The above sampling can be random sampling. Random sampling can be understood as the sampling location and / or the sampling size being randomly determined.

[0182] The first image block and the second image block can be rectangular images.

[0183] 603. Based on the fact that the intersection ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold, feature extraction is performed on the first image block and the second image block respectively through a feature extraction network to obtain a first feature map and a second feature map.

[0184] In this context, there may be a common region (overlapping region) between the region where the first image block and the region where the second image block are located in the sample image. The ratio of the area of ​​this common region to the total area occupied by the first and second image blocks in the sample image is the intersection-over-union (IoU) ratio between the regions where the first and second image blocks are located in the sample image (for example, refer to...). Figure 7 (As shown).

[0185] In one possible implementation, the threshold can be a value greater than or equal to 0.4.

[0186] The fundamental reason for global inconsistency in multi-instance scenarios lies in the fact that randomly sliced ​​image patches may represent completely different scenes and semantic information. However, if two image patches can be controlled to be sufficiently "close" to each other, then based on image continuity within a local region, the two image patches can be considered to describe similar semantic information, thereby restoring the basic assumption of contrastive learning. In the embodiments of this application, the intersection-union ratio (IUGR) between two image patches can be used as a way to measure distance. Here, IUGR refers to the ratio of the overlapping area of ​​two image patches to the total coverage area (see...). Figure 7 (Left). The requirement is that subsequent feature extraction and similarity calculation will only be performed if the intersection-union ratio (IU) of two randomly generated image patches is greater than a given threshold (see...). Figure 7 right).

[0187] It should be understood that the intersection-union ratio (IU) threshold should not be set too low, but it is also not better to be higher, because we do not want the image patches to be completely unrelated, but we also do not want them to be completely identical. Therefore, the choice of the IU threshold is actually controlling the balance between data noise and data complexity in multi-instance self-supervised learning.

[0188] For example, refer to Figure 8 , Figure 8 An example of a sample image is shown for reference. Figure 9 , Figure 9 This is an illustration of the first and second image blocks. Figure 9 The Intersection over Union (IOU) ratio between the first and second image blocks shown is 0.3, as referenced. Figure 10 , Figure 10 This is an illustration of the first and second image blocks. Figure 10 The Intersection over Union (IOU) ratio between the first and second image blocks shown is 0.5, as referenced. Figure 11 , Figure 11 This is an illustration of the first and second image blocks. Figure 11 The intersection-over-union (IOU) ratio between the first and second image blocks shown is 0.7.

[0189] In this embodiment of the application, based on the fact that the intersection ratio between the first image block and the second image block in the region where they are located on the sample image is greater than a threshold, feature extraction networks can be used to extract features from the first image block and the second image block respectively to obtain a first feature map and a second feature map.

[0190] The feature extraction network in the embodiments of this application will be introduced next:

[0191] In one possible implementation, the feature extraction network can be a backbone network, which receives the input image (e.g., the first image block and the second image block in the embodiments of this application) and performs convolution processing on the input image to generate multiple feature maps.

[0192] It should be noted that "convolution processing of the input image" here should not be understood as simply performing convolution processing on the input image. In some implementations, convolution processing and other processing can be performed on the input image.

[0193] It should be noted that "convolution processing of the first image to generate multiple feature maps" should not be understood as simply performing multiple convolution processes on the image, with each convolution process generating one feature map. In other words, it should not be understood as each feature map being obtained based on convolution processing of the image. Rather, from an overall perspective, the image is the source of multiple feature maps. In one implementation, convolution processing of the image can be performed to obtain one feature map, and then convolution processing can be performed on the generated feature map to obtain another feature map, and so on, to obtain multiple feature maps.

[0194] It should be noted that a series of convolution processes can be performed on the input image. Specifically, in each convolution process, the feature map obtained from the previous convolution process can be convolved to obtain a feature map. Multiple feature maps can be obtained in this way.

[0195] It should be noted that multiple feature maps can be feature maps with multi-scale resolution, that is, multiple feature maps are not feature maps with the same resolution. In one optional implementation, multiple feature maps can form a feature pyramid.

[0196] See Figure 12 , Figure 12 This is a schematic diagram of the structure of a feature extraction network provided in an embodiment of this application. Figure 12 As shown, the backbone network is used to receive the input image and perform convolution processing on the input image, outputting feature maps (feature map C1, feature map C2, feature map C3, and feature map C4) with different resolutions corresponding to the image; that is, it outputs feature maps of different sizes corresponding to the image. The backbone network completes the extraction of basic features and provides corresponding features for subsequent detection.

[0197] Specifically, the backbone network performs a series of convolutional processes on the input image to obtain feature maps at different scales (with different resolutions). These feature maps provide the basic features for subsequent detection modules. The backbone network can take various forms, such as the visual geometry group (VGG), the residual neural network (ResNet), and the core structure of GoogLeNet (Inception-net).

[0198] The backbone network can perform convolution processing on the input image to generate several convolutional feature maps of different scales. Each feature map is an H*W*C matrix, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map.

[0199] The backbone can adopt a variety of existing convolutional network frameworks, such as VGG16, ResNet50, and Inception-Net. The following explanation uses ResNet18 as the backbone.

[0200] Assume the input image has a resolution of H*W*3 (height H, width W, and 3 channels, i.e., three RGB channels). The input image can be convolved through a ResNet18 convolutional layer, Res18-Conv1, to generate a feature map C1. This feature map is downsampled twice relative to the input image, and the number of channels is increased to 64, so the resolution of C1 is H / 4*W / 4*64. C1 can then be convolved through ResNet18's Res18-Conv2 to obtain feature map C2, which has the same resolution as C1. C2 is further convolved through Res18-Conv3 to generate feature map C3, which is further downsampled compared to C2, doubling the number of channels, and has a resolution of H / 8*W / 8*128. Finally, C3 is convolved through Res18-Conv4 to generate feature map C4, with a resolution of H / 16*W / 16*256.

[0201] It should be noted that the backbone network in the embodiments of this application can also be called the backbone network, and this is not a limitation.

[0202] It should be noted that, Figure 12 The backbone network shown is only one implementation method and does not constitute a limitation of this application.

[0203] In this embodiment of the application, feature extraction is performed on the first image block and the second image block by a feature extraction network to obtain a first feature map and a second feature map. The first feature map can be multiple feature maps (feature pyramids) obtained by processing the first image block through the feature extraction network, or it can be a feature map obtained after a certain convolution operation. Similarly, the second feature map can be multiple feature maps (feature pyramids) obtained by processing the second image block through the feature extraction network, or it can be a feature map obtained after a certain convolution operation.

[0204] The first feature map and the second feature map are the same size.

[0205] 604. Determine the loss based on the difference between the first feature map and the second feature map, and update the feature extraction network based on the loss to obtain the updated feature extraction network.

[0206] In this embodiment of the application, after feature extraction is performed on the first image block and the second image block respectively by a feature extraction network to obtain a first feature map and a second feature map based on the fact that the intersection ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold, a loss can be determined based on the difference between the first feature map and the second feature map, and the feature extraction network can be updated based on the loss to obtain an updated feature extraction network.

[0207] In one possible implementation, the first image block and the second image block can be aligned first to obtain the aligned first image block and the aligned second image block.

[0208] In one implementation, the sample image may include a target region, which is the overlapping area where the first image block and the second image block are located on the sample image. Based on the target region, a first sub-feature map corresponding to the target region in the first feature map and a second sub-feature map corresponding to the target region in the second feature map are determined. The first sub-feature map is upsampled to obtain the aligned first image block. The second sub-feature map is upsampled to obtain the aligned second image block, wherein the aligned first image block and the aligned second image block have the same size.

[0209] To distinguish multi-instance features, the global pooling layer after the backbone network needs to be discarded to maintain the two-dimensional structure and positional information of the feature map. However, this also introduces the additional problem of feature misalignment, meaning that the two-dimensional feature map no longer has a one-to-one correspondence of the same relative positions. This application provides two different methods for feature alignment: Region of Interest (ROI) alignment, which uses the overlapping parts of image patches as the ROIs of the two image patches respectively. For example, RoI Align can be used to extract only the features of the overlapping parts for subsequent calculations. This method is intuitive but does not fully utilize the information of the non-overlapping parts (e.g., refer to...). Figure 13 (As shown).

[0210] In one implementation, the first feature map and the second feature map are of the same size. The first feature map includes M first feature points, and the second feature map includes M second feature points. The M first feature points correspond to M first pixels in the sample image, and the M second feature points correspond to M second pixels in the sample image. There is a one-to-one correspondence between the M first pixels and the M second pixels. A third feature map can be obtained based on the M first pixels and the M second pixels. The third feature map is of the same size as the second feature map and includes M third feature points, each of which is obtained based on the pixel position difference between corresponding first and second pixels. The third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block. Optionally, the third feature map and the first feature map can be stitched together in the depth direction.

[0211] In this process, displacement alignment involves taking each pair of pixels at the same relative position, calculating their coordinate displacement in the original image, and concatenating this displacement with the feature map (by stitching them together in the depth direction). This concatenation is provided as additional lateral information to the prediction network for implicit feature alignment, aiding subsequent feature prediction (e.g., referring to...). Figure 14 As shown in the figure, it makes full use of the feature information of non-overlapping regions, and at the same time makes the subsequent similarity measurement more flexible.

[0212] After obtaining the aligned first feature map and the aligned second feature map, the loss can be determined based on the difference between the aligned first feature map and the aligned second feature map.

[0213] The following describes how to obtain the difference between the aligned first feature map and the aligned second feature map:

[0214] In one possible implementation, the first feature map can be processed by a target prediction network to obtain a predicted value for each first feature point; wherein the target prediction network may contain convolution operations (e.g., 1*1 convolution kernels).

[0215] The step described above, which processes the M first feature points of the first feature map using a target prediction network, can be called the online branch. The online branch then predicts the clustering center results of the target branch using the prediction network. Optionally, the online branch can also deploy an additional spatial self-attention module to fully consider contextual information and make more accurate predictions. Specifically, if the input to the online branch is R, then the final prediction result Q of the online branch is:

[0216]

[0217] Where H and W are the length and width of R and R′, respectively, q θ (·) is the prediction network, and sim(·,·) is defined as:

[0218] sim(R i,j ,R i′,j′ )=(max(cos(R i,j ,R i′,j′ ),0)) 2 ;

[0219] In one possible implementation, the M second feature points of the first feature map can be clustered based on a target clustering algorithm to update the feature values ​​of the M second feature points, wherein the feature value of each updated second feature point is the cluster center feature value of the cluster category it belongs to.

[0220] The step of clustering the M second feature points of the first feature map based on the target clustering algorithm can be called the target branch. The target clustering algorithm can be, but is not limited to, K-means, mean-shift clustering, density-based clustering methods, agglomerative hierarchical clustering, graph community detection, Gaussian mixture model k-means (GMM k-means), and other clustering methods.

[0221] In multi-instance scenarios, there exists a hierarchical clustering structure, including category clustering, instance clustering, and pixel clustering from top to bottom. Furthermore, the definition of clustering is a relative concept; that is, the same pixel may belong to different clusters in image patches within different contexts (see...). Figure 15 The two image patches shown in the lower right corner have pixel P belonging to the "human" cluster in the left patch and the "man" cluster in the right patch. Based on this observation, intra-image clustering can be used to distinguish the features of different instances, making the features of pixels within the same cluster as similar as possible, while making the features of pixels in different clusters as dissimilar as possible, thus promoting consistency in the clustering analysis results of the two image patches. A specific approach can be as follows... Figure 16 As shown, optionally, K-means clustering can be used on the target branch to obtain the cluster center (mean, smooth) feature of each point.

[0222] In this embodiment, image clustering is used, which enables the network to distinguish features of different instances. On the other hand, by considering the overall information of the two-dimensional feature map, the regression target provided by the target branch is more robust, while the online branch introduces a global perspective by deploying a self-attention module to obtain more accurate prediction results.

[0223] After obtaining the predicted value of each first feature point and the feature value of each updated second feature point, the loss can be determined based on the difference between the predicted value of each first feature point and the feature value of each updated second feature point. This loss can represent the difference (or be described as similarity) between the predicted value of each first feature point and the feature value of each updated second feature point. The two branches utilize the overall information of the two-dimensional feature maps in different ways to better measure the similarity between the two two-dimensional feature maps.

[0224] In one implementation, similarity can be defined as:

[0225]

[0226] Where the inputs for the online branch and the target branch are R and R', respectively, and Q i,j For the output of the online branch, Kmeans(R′) i,j ) is the output of the target branch.

[0227] After obtaining the loss, the online branches (e.g., including the target prediction network and the feature extraction network) can be updated based on the loss. For example, gradient descent can be used to update the parameters of the online branches, while the target branch parameters are updated to a moving average of the online branch parameters. This training process is iterated until the network converges.

[0228] Reference Figure 17 and Figure 18 , Figure 17 and Figure 18 A multi-instance self-supervised learning framework (MultiSiam) data augmentation, feature extraction, and network training process is provided for embodiments of this application:

[0229] For each input image, two image patches are randomly cut. The intersection-union (IU) ratio of the two patches is checked against a given threshold. If not, the patching is repeated until the IU threshold is exceeded. If it is, scale adjustment and color / texture enhancement can continue. Two-dimensional feature maps of the two image patches are extracted using a backbone network. A projection network is used to reduce the dimensionality of the two image patch feature maps to reduce subsequent computation. A feature alignment module is used to restore the one-to-one correspondence at the same relative positions in the two-dimensional feature maps using region-of-interest alignment or displacement alignment. Note that the feature alignment module does not change the spatial scale of the input feature maps. (The image is then displayed in the target branch (left) feature map.) The K-means clustering algorithm is run to obtain the clustering result of each pixel and the corresponding cluster center features. To prevent network degradation, a gradient truncation operation is added at the end of the target branch. The online branch (right) predicts the clustering result of the target branch through the self-attention module and the prediction network. The two-dimensional clustering similarity is calculated according to the definition and weighted and summed with the one-dimensional feature similarity of the step (for example, the weights of both can be 0.5 by default) to obtain the final feature similarity measure. The parameters of the online branch are updated using gradient descent, and the parameters of the target branch are updated to the moving average of the parameters of the online branch. The iterative operation continues until the network converges.

[0230] It should be understood that the above process of training the feature extraction network can be considered a pre-training process. After obtaining the updated feature extraction network, it can be fine-tuned for the target task by combining it with the downstream task dataset.

[0231] For example, the updated feature extraction network can be connected to the downstream task network, and the updated feature extraction network and the downstream task network can be fine-tuned based on the downstream task dataset to obtain the target network. The target network can be deployed on the execution device for the feedforward process.

[0232] For example, taking object detection as an example of a downstream task, the downstream task network may include one or more heads, such as Figure 19 As shown, one or more heads are used to detect task objects in a task based on the feature map output by the feature extraction network, and output 2D bounding boxes of the regions where the task objects are located and the confidence scores of each 2D bounding box; optionally, multiple heads can be set up in parallel, and each head can perform the detection of different task objects; wherein, the task object is the object that needs to be detected in the task; the higher the confidence score, the greater the probability that the object corresponding to the task exists in the 2D bounding box corresponding to the confidence score.

[0233] In this embodiment of the application, different heads can perform different 2D detection tasks. For example, one of the multiple heads can perform vehicle detection and output 2D bounding boxes and confidence scores for Car / Truck / Bus; head1 of the multiple heads can perform human detection and output 2D bounding boxes and confidence scores for Pedestrian / Cyclist / Tricyle; and head of the multiple heads can perform traffic light detection and output 2D bounding boxes and confidence scores for Red_Trafficligh / Green_Trafficlight / Yellow_TrafficLight / Black_TrafficLight.

[0234] In this embodiment, the downstream task network may include multiple serial heads; each serial head is connected to a parallel head. It should be emphasized that, in practice, serial heads are not necessary. For scenarios that only require the detection of 2D bounding boxes, serial heads are not required.

[0235] The serial head can be used to: extract features of the region containing the 2D bounding box of the task object provided by the parallel head it is connected to, on one or more feature maps on the feature extraction network; and predict the 3D information, mask information, or keypoint information of the task object based on the features of the region containing the 2D bounding box. The serial head can optionally be concatenated after the parallel head to perform 3D / mask / keypoint detection of objects inside the 2D bounding box, based on the detected 2D bounding box. For example, serial 3D_head0 estimates the vehicle's orientation, centroid, and length, width, and height, thus outputting the vehicle's 3D bounding box; serial Mask_head0 predicts the vehicle's fine mask, thus segmenting the vehicle; and serial Keypoint_head0 estimates the vehicle's keypoints. The serial head is not mandatory. Some tasks do not require 3D / mask / keypoint detection, so a serial head is not needed. For example, traffic light detection only requires detecting 2D bounding boxes, so a serial head is not required. In addition, some tasks can choose to cascade one or more serial heads depending on the specific requirements of the task. For example, parking lot detection requires not only 2D bounding boxes but also key points of parking spaces. Therefore, in this task, only one serial keypoint head needs to be cascaded, and 3D and mask heads are not required.

[0236] In this embodiment, the header is connected to a feature extraction network. Based on the feature map provided by the feature extraction network, the header can detect 2D bounding boxes for a task and output the 2D bounding boxes of the objects in that task, along with their corresponding confidence scores, etc. The following describes a schematic diagram of the header structure, referring to... Figure 20 , Figure 20 This is an illustration of a header, such as... Figure 20 As shown in the diagram, the head includes three modules: Region Proposal Network (RPN), ROI-ALIGN, and RCNN.

[0237] The RPN module can be used to predict the region where the task object is located on one or more third feature maps provided by the feature extraction network, and output candidate 2D boxes matching the region; or it can be understood that the RPN predicts regions where the task object may exist on one or more horizontal maps output by the feature extraction network, and provides bounding boxes for these regions, which are called candidate regions (Proposals). For example, when the head is responsible for detecting cars, its RPN layer predicts candidate boxes where cars may exist; when the head is responsible for detecting people, its RPN layer predicts candidate boxes where people may exist. Of course, these proposals are not accurate; on the one hand, they may not contain the task object, and on the other hand, these boxes are not tight.

[0238] The 2D candidate region prediction process can be implemented by the RPN module of the head, which predicts regions where the task object may exist based on the feature map provided by the feature extraction network, and provides candidate boxes (also called candidate regions, proposals) for these regions. In this embodiment, if the head is responsible for detecting vehicles, its RPN layer predicts candidate boxes where vehicles may exist.

[0239] The RPN layer may generate a feature map RPN Hidden on the third feature map provided by the feature extraction network through, for example, a 3*3 convolution. The subsequent RPN layer in the head will predict Proposals from RPN Hidden. Specifically, the RPN layer in the head predicts the coordinates and confidence of the Proposal at each position of RPN Hidden through a 1*1 convolution respectively. A higher confidence indicates a higher probability that the Proposal contains an object for the task. For example, a higher score of a certain Proposal in the head indicates a higher probability that it contains a vehicle. The Proposals predicted by each RPN layer need to pass through a Proposal merging module, which removes redundant Proposals according to the degree of overlap between the Proposals (this process may adopt, but is not limited to, the NMS algorithm), and selects N (N<K) Proposals with the highest scores from the remaining K Proposals as candidate regions where an object may exist. These Proposals are inaccurate: on the one hand, they do not necessarily contain the object for the task, and on the other hand, these frames are not compact. Therefore, the RPN module is only a coarse detection process, and the subsequent RCNN module is required for refinement. When the RPN module regresses the coordinates of the Proposals, it does not directly regress the absolute values of the coordinates, but regresses the coordinates relative to the Anchors. The higher the match between these Anchors and the actual object, the higher the probability that the RPN can detect the object.

[0240] The ROI-ALIGN module is configured to extract the features of the region where the candidate 2D frame is located from a feature map provided by the feature extraction network according to the region predicted by the RPN module; that is, the ROI-ALIGN module mainly extracts the features of the region where each Proposal is located on a certain feature map according to the Proposals provided by the RPN module, and resizes them to a fixed size to obtain the feature of each Proposal. It can be understood that the ROI-ALIGN module may use, but is not limited to, feature extraction methods such as ROI-POOLING (Region of Interest Pooling) / ROI-ALIGN (Region of Interest Alignment) / PS-ROIPOOLING (Position-Sensitive Region of Interest Pooling) / PS-ROIALIGN (Position-Sensitive Region of Interest Alignment).

[0241] The RCNN module performs convolutional processing on the features of the regions containing the candidate 2D bounding boxes using a neural network to obtain the confidence scores of the candidate 2D bounding boxes belonging to each object category. It then adjusts the coordinates of the 2D bounding boxes in the candidate regions using the neural network, making the adjusted 2D bounding boxes better match the shape of the actual objects than the original candidate 2D bounding boxes. Adjusted 2D bounding boxes with confidence scores greater than a preset threshold are selected as the 2D bounding boxes for the region. In other words, the RCNN module mainly refines the features of each proposal proposed by the ROI-ALIGN module, obtaining the confidence scores of each proposal belonging to each category (e.g., for the car task, it provides four scores: Background / Car / Truck / Bus). Simultaneously, it adjusts the coordinates of the 2D bounding boxes of the proposals, outputting more compact 2D bounding boxes. These 2D bounding boxes are then merged using nonmaximum suppression (NMS) to become the final 2D bounding box output.

[0242] 2D candidate region fine classification is mainly composed of Figure 20 The RCNN module in the head of the algorithm implements a more compact 2D bounding box coordinate system based on the features extracted from each proposal by the ROI-ALIGN module. It also classifies the proposal and outputs the confidence score for each category. RCNN can be implemented in many ways. The ROI-ALIGN module outputs N*14*14*256 features of proposals. In the RCNN module, these features are first processed by ResNet18 convolutional module 4 (Res18-Conv5), resulting in N*7*7*512 features. Then, a Global Avg Pooling layer averages the 7*7 features in each channel of the input features to obtain N*512 features, where each 1*512 feature vector represents the features of each proposal. Next, two fully connected layers (FC) are used to regress the precise coordinates of the bounding boxes (outputting an N*4 vector, where these four values ​​represent the x / y coordinates of the box's center point, and the box's width and height), and the confidence score of the box's category (in head0, the score for whether the box is Background / Car / Truck / Bus needs to be given). Finally, a box merging operation is used to select the boxes with the highest scores, and NMS is used to remove duplicate boxes, resulting in a compact bounding box output.

[0243] In some practical applications, this perceptual network can also include other heads, which can further perform 3D / mask / keypoint detection based on the detected 2D bounding boxes. For example, taking 3D as an example, the ROI-ALIGN module extracts the features of the region where each 2D bounding box is located on the feature map output by the feature extraction network based on the accurate 2D bounding boxes provided by the head. Assuming that the number of 2D bounding boxes is M, then the feature size output by the ROI-ALIGN module is M*14*14*256. It is first processed by Res18-Conv5 of ResNet18, and the output feature size is N*7*7*512. Then it is processed by a Global AvgPool (average pooling layer) to average the 7*7 features of each channel in the input features to obtain M*512 features, where each 1*512-dimensional feature vector represents the feature of each 2D bounding box. Next, three fully connected layers (FC) are used to regress the orientation (M*1 vector), centroid coordinates (M*2 vector, these two values ​​represent the x / y coordinates of the centroid), and length, width, and height of the object in the bounding box.

[0244] It should be noted that, Figure 19 and Figure 20 The header shown is only one implementation method and does not constitute a limitation of this application.

[0245] Taking two large-scale multi-instance autonomous driving datasets as upstream pre-training datasets as examples, in terms of experimental data selection, upstream self-supervised pre-training can use multi-instance autonomous driving datasets including the Waymo public dataset and the SODA-5M dataset. The Waymo public dataset contains nearly 790,000 unlabeled images with image scales ranging from (1920, 968) to (1920, 1280), making it the largest open-source autonomous driving dataset to date; the SODA-5M dataset contains 5 million high-quality autonomous driving images, all with an image scale of (1920, 1280). Downstream transfer datasets mainly consider two widely used autonomous driving semantic segmentation benchmark tasks: Cityscapes and BDD100K. Cityscapes contains annotations for 8 semantic categories, with 2975 training set images and 500 validation set images collected from 27 cities; BDD100K contains annotations for 9 semantic categories, with 70,000 training set images and 10,000 validation set images collected from 4 cities.

[0246] Since the upstream dataset lacks manual annotations, making it impossible to directly evaluate algorithm performance, a self-supervised pre-trained model is used to initialize the backbone network for the downstream task, followed by fine-tuning. The mean Intersection over Union (mIOU) of the prediction results on the downstream task validation set is used as the evaluation metric to judge the quality of the pre-trained model, where a higher mIOU value is better. First, self-supervised pre-training is performed on the upstream autonomous driving dataset according to the proposed framework. After the network converges, upstream task-specific modules such as the projection network and prediction network are discarded, and only the backbone network parameters are transferred to the downstream task. The network parameters are then fine-tuned using the downstream task training set. The fine-tuned model is then used to make predictions on the validation set, and the mIOU of the final prediction results is reported as the evaluation metric.

[0247] ResNet-50 was used as the default backbone network. To better adapt to large-batch training, the LARS optimizer and cosine learning rate decay were used to stabilize the network training process. To better compare the pre-training results of the model on different datasets, we kept the GPU time consistent across different datasets. Specifically, we performed 325 epochs and 55 epochs of pre-training on Waymo and SODA-5M respectively to ensure a fair comparison with 200 epochs of pre-training on ImageNet.

[0248] By pre-training on the publicly available autonomous driving dataset Waymo, the method proposed in this application successfully achieved significant performance improvements of 4.7% on Cityscapes and 3.9% on BDD100K compared to the benchmark model BYOL, achieving state-of-the-art results in Waymo pre-training. This demonstrates that this application can better utilize multi-instance datasets for feature learning. However, considering that Waymo has fewer images than ImageNet (0.79M vs. 1.28M) and exhibits significant foreground-background imbalance, Waymo is actually inferior to ImageNet in both image quantity and quality, making it difficult to surpass ImageNet pre-training results. Therefore, the SODA-5M autonomous driving dataset was additionally used for pre-training, successfully exceeding the transfer results of ImageNet pre-training and demonstrating the effectiveness of domain-specific pre-training.

[0249] It is worth noting that due to the single-sample assumption, collecting more images similar to the ImageNet dataset inevitably requires manual data filtering and cleaning. In contrast, the multi-instance scenario eliminates the single-instance assumption and can collect a very large-scale pre-training dataset at a very low cost. Therefore, multi-instance self-supervised learning is more practical for industrial-grade datasets.

[0250] Referring to Table 1, which shows the pre-training results of the embodiments of this application on a multi-instance autonomous driving dataset, the embodiments of this application achieve significant performance improvement compared to the benchmark model; when using a very large-scale autonomous driving dataset for pre-training, they successfully outperform ImageNet pre-training, demonstrating the effectiveness of domain-specific pre-training.

[0251] Table 1

[0252]

[0253] For example, the finally learned features are visualized and analyzed. K-means clustering analysis is performed on the last two-dimensional feature map of the pre-trained backbone network. The clustering results are as follows: Figure 21 As shown, compared with the prior art, the embodiments of this application can not only judge the information of the foreground and background well, but also accurately distinguish the features of different instances. At the same time, the feature clustering results of the embodiments of this application are smoother, which also means that the translation invariance and feature robustness are enhanced in the learned feature representation.

[0254] Furthermore, the scalability of this framework is verified by deploying the solution provided in this application embodiment on a single instance for self-supervised learning. Except for the experimental data, all other settings are consistent with Technical Solution 1. Regarding the selection of experimental data, the upstream dataset can use the ImageNet dataset with the single-instance assumption, containing natural images of 1000 categories, totaling 1.28 million training images; here, manual annotation can be omitted, and self-supervised representation learning can be performed directly using image data. For the downstream dataset, two additional general detection datasets, VOC and COCO, are considered. The VOC dataset contains 20 semantic category annotations, with 10,000 training images and 4900 test images; the COCO dataset contains 80 semantic category annotations, with approximately 118,000 training images and 5000 validation images.

[0255] Verification has shown that although the embodiments of this application are designed for multi-instance scenarios, they can still effectively utilize single-instance scene images to obtain high-quality feature representations and achieve the best transfer results on multiple downstream tasks (see Table 2). This demonstrates the versatility and scalability of the framework of the embodiments of this application.

[0256] Table 2

[0257]

[0258] This application provides a model training method, comprising: acquiring a sample image; sampling the sample image to obtain a first image patch and a second image patch, wherein the first image patch and the second image patch are different image patches on the sample image; based on the intersection-over-union ratio (IoU) between the regions where the first image patch and the second image patch are located on the sample image being greater than a threshold, performing feature extraction on the first image patch and the second image patch respectively through a feature extraction network to obtain a first feature map and a second feature map; determining a loss based on the difference between the first feature map and the second feature map, and updating the feature extraction network based on the loss to obtain an updated feature extraction network. Using IoU as a measure of the relative distance between two image patches, by requiring the IoU of image patches to be greater than a given threshold to ensure semantic consistency within a local region to restore the basic assumption of contrastive learning, the image patch is restricted to a local region, thereby transforming global inconsistency into local consistency, and thus improving the data processing accuracy of the updated feature extraction network. Simultaneously, the selection of the threshold can effectively control the balance between data noise and data complexity in different scenarios.

[0259] Reference Figure 22 , Figure 22 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application, as shown below. Figure 22 As shown, the apparatus 2200 provided in this application embodiment includes:

[0260] The acquisition module 2201 is used to acquire sample images.

[0261] The specific description of the acquisition module 2201 can be found in the description of step 601 in the above embodiments, and will not be repeated here.

[0262] The sampling module 2202 is used to sample the sample image to obtain a first image block and a second image block, wherein the first image block and the second image block are different image blocks on the sample image;

[0263] The specific description of the sampling module 2202 can be found in the description of step 602 in the above embodiment, and will not be repeated here.

[0264] The feature extraction module 2203 is used to extract features from the first image block and the second image block respectively through a feature extraction network based on the fact that the intersection ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold, so as to obtain a first feature map and a second feature map.

[0265] The specific description of the feature extraction module 2203 can be found in the description of step 603 in the above embodiment, and will not be repeated here.

[0266] The model update module 2204 is used to determine the loss based on the difference between the first feature map and the second feature map, and update the feature extraction network based on the loss to obtain the updated feature extraction network.

[0267] The specific description of the model update module 2204 can be found in the description of step 604 in the above embodiment, and will not be repeated here.

[0268] In one possible implementation, the threshold is a value greater than or equal to 0.4.

[0269] In one possible implementation, the device further includes:

[0270] An alignment module is used to align the first image block and the second image block before determining the loss based on the difference between the first feature map and the second feature map, so as to obtain an aligned first image block and an aligned second image block.

[0271] The model update module is specifically used for:

[0272] The loss is determined based on the difference between the aligned first feature map and the aligned second feature map.

[0273] In one possible implementation, the sample image includes a target region, which is the overlapping area where the first image block and the second image block are located on the sample image. The alignment module is specifically used for:

[0274] Based on the target region, a first sub-feature map corresponding to the target region in the first feature map and a second sub-feature map corresponding to the target region in the second feature map are determined.

[0275] The first sub-feature map is upsampled to obtain the aligned first image block;

[0276] The second sub-feature map is upsampled to obtain the aligned second image block, wherein the aligned first image block has the same size as the aligned second image block.

[0277] In one possible implementation, the first feature map and the second feature map have the same size. The first feature map includes M first feature points, and the second feature map includes M second feature points. The M first feature points correspond to M first pixels in the sample image, and the M second feature points correspond to M second pixels in the sample image. There is a one-to-one correspondence between the M first pixels and the M second pixels. The alignment module is specifically used for:

[0278] A third feature map is obtained based on the M first pixels and the M second pixels. The third feature map and the second feature map have the same size. The third feature map includes M third feature points, each of which is obtained based on the pixel position difference between the corresponding first pixels and second pixels.

[0279] The third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block.

[0280] In one possible implementation, the alignment module is specifically used for:

[0281] The third feature map is stitched together with the first feature map in the depth direction.

[0282] In one possible implementation, the model update module is specifically used for:

[0283] The target prediction network processes the M first feature points of the first feature map to obtain the predicted value of each first feature point.

[0284] Based on the target clustering algorithm, the M second feature points of the first feature map are clustered to update the feature values ​​of the M second feature points, wherein the feature value of each updated second feature point is the cluster center feature value of the cluster category in which it belongs.

[0285] The loss is determined based on the difference between the predicted value of each first feature point and the feature value of each updated second feature point.

[0286] In one possible implementation, the model update module is further configured to:

[0287] The target prediction network is updated based on the loss.

[0288] In one possible implementation, the sample image includes multiple objects, including at least one of people, vehicles, traffic signs, lane lines, and plants.

[0289] In one possible implementation, the device further includes:

[0290] The data processing module is used to acquire the target network and the image to be processed, wherein the target network includes the updated feature extraction network and the downstream task network;

[0291] The target network is used to process the image to be processed in order to obtain the processing result.

[0292] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 23 , Figure 23 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 2300 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device, or a server, etc., and is not limited thereto. Specifically, the execution device 2300 includes: a receiver 2301, a transmitter 2302, a processor 2303, and a memory 2304 (wherein the execution device 2300 may have one or more processors 2303). Figure 23 (Taking a processor as an example), the processor 2303 may include an application processor 23031 and a communication processor 23032. In some embodiments of this application, the receiver 2301, transmitter 2302, processor 2303, and memory 2304 may be connected via a bus or other means.

[0293] Memory 2304 may include read-only memory and random access memory, and provides instructions and data to processor 2303. A portion of memory 2304 may also include non-volatile random access memory (NVRAM). Memory 2304 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0294] The processor 2303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0295] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 2303. The processor 2303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 2303 or by instructions in software form. The processor 2303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 2303 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 2304. Processor 2303 reads the information in memory 2304 and, in conjunction with its hardware, completes the steps of the above method.

[0296] Receiver 2301 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 2302 can be used to output digital or character information through the first interface; transmitter 2302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 2302 may also include a display device such as a display screen.

[0297] In one embodiment of this application, the processor 2303 is configured to execute the above-described... Figure 6 The model is obtained by the model training method.

[0298] This application also provides a training device; please refer to [link / reference]. Figure 24 , Figure 24 This is a schematic diagram of a training device provided in an embodiment of this application. The training device 2400 can be equipped with... Figure 22 The image training apparatus described in the corresponding embodiment is used to implement Figure 22In accordance with the function of the image training device in the corresponding embodiment, specifically, the training device 2400 is implemented by one or more servers. The training device 2400 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 2424 (e.g., one or more processors) and memory 2432, and one or more storage media 2430 (e.g., one or more mass storage devices) for storing application programs 2442 or data 2444. The memory 2432 and storage media 2430 can be temporary or persistent storage. The program stored in the storage media 2430 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the training device. Furthermore, the CPU 2424 may be configured to communicate with the storage media 2430 and execute the series of instruction operations in the storage media 2430 on the training device 2400.

[0299] The training device 2400 may also include one or more power supplies 2426, one or more wired or wireless network interfaces 2450, one or more input / output interfaces 2458; or, one or more operating systems 2441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0300] In this embodiment of the application, the central processing unit 2424 is used to execute the above-described... Figure 6 The model training method.

[0301] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0302] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0303] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0304] For details, please refer to Figure 25 , Figure 25 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 2500. The NPU 2500 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 2503, which is controlled by the controller 2504 to retrieve matrix data from the memory and perform multiplication operations.

[0305] In some implementations, the arithmetic circuit 2503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 2503 is a two-dimensional pulsating array. The arithmetic circuit 2503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2503 is a general-purpose matrix processor.

[0306] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2508.

[0307] Unified memory 2506 is used to store input and output data. Weight data is directly transferred to weight memory 2502 via Direct Memory Access Controller (DMAC) 2505. Input data is also transferred to unified memory 2506 via DMAC.

[0308] BIU stands for Bus Interface Unit 2510, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 2509.

[0309] The Bus Interface Unit (BIU) 2510 is used by the instruction fetch memory 2509 to fetch instructions from external memory, and also by the memory access controller 2505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0310] The DMAC is mainly used to move input data from external memory DDR to unified memory 2506, or to weight data to weight memory 2502, or to input data to input memory 2501.

[0311] The vector computation unit 2507 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0312] In some implementations, the vector computation unit 2507 can store the processed output vector in the unified memory 2506. For example, the vector computation unit 2507 can apply a linear function, or a nonlinear function, to the output of the computation circuit 2503, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 2507 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 2503, for example, for use in subsequent layers of the neural network.

[0313] The instruction fetch buffer 2509 connected to the controller 2504 is used to store the instructions used by the controller 2504.

[0314] Unified memory 2506, input memory 2501, weighted memory 2502, and instruction fetch memory 2509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0315] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0316] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0317] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0318] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0319] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A model training method, characterized in that, The method includes: Acquire a sample image, which includes multiple objects; The sample image is sampled to obtain a first image block and a second image block, wherein the first image block and the second image block are different image blocks on the sample image; Based on the fact that the intersection ratio between the regions where the first image block and the second image block are located in the sample image is greater than a threshold, feature extraction networks are used to extract features from the first image block and the second image block respectively to obtain a first feature map and a second feature map, wherein the first feature map and the second feature map are two-dimensional feature maps; Based on the difference between the first feature map and the second feature map, a loss is determined, and the feature extraction network is updated based on the loss to obtain an updated feature extraction network.

2. The method according to claim 1, characterized in that, The threshold is a value greater than or equal to 0.

4.

3. The method according to claim 1 or 2, characterized in that, Before determining the loss based on the difference between the first feature map and the second feature map, the method further includes: The first image block and the second image block are aligned to obtain an aligned first image block and an aligned second image block; The step of determining the loss based on the difference between the first feature map and the second feature map includes: The loss is determined based on the difference between the aligned first feature map and the aligned second feature map.

4. The method according to claim 3, characterized in that, The sample image includes a target region, which is the overlapping area where the first image block and the second image block are located on the sample image. Aligning the first image block and the second image block includes: Based on the target region, a first sub-feature map corresponding to the target region in the first feature map and a second sub-feature map corresponding to the target region in the second feature map are determined. The first sub-feature map is upsampled to obtain the aligned first image block; The second sub-feature map is upsampled to obtain the aligned second image block, wherein the aligned first image block has the same size as the aligned second image block.

5. The method according to claim 3, characterized in that, The first feature map and the second feature map are of the same size. The first feature map includes M first feature points, and the second feature map includes M second feature points. The M first feature points correspond to M first pixels in the sample image, and the M second feature points correspond to M second pixels in the sample image. There is a one-to-one correspondence between the M first pixels and the M second pixels. The method further includes: A third feature map is obtained based on the M first pixels and the M second pixels. The third feature map and the second feature map have the same size. The third feature map includes M third feature points, each of which is obtained based on the pixel position difference between the corresponding first pixels and second pixels. The third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block.

6. The method according to claim 5, characterized in that, The step of fusing the third feature map with the first feature map includes: The third feature map is stitched together with the first feature map in the depth direction.

7. The method according to claim 1, characterized in that, The step of determining the loss based on the difference between the first feature map and the second feature map includes: The target prediction network processes the M first feature points of the first feature map to obtain the predicted value of each first feature point. Based on the target clustering algorithm, the M second feature points of the first feature map are clustered to update the feature values ​​of the M second feature points, wherein the feature value of each updated second feature point is the cluster center feature value of the cluster category in which it belongs. The loss is determined based on the difference between the predicted value of each first feature point and the feature value of each updated second feature point.

8. The method according to claim 7, characterized in that, The method further includes: The target prediction network is updated based on the loss.

9. The method according to claim 1, characterized in that, The objects include at least one of people, vehicles, traffic signs, lane lines, and plants.

10. The method according to claim 1, characterized in that, The method further includes: Acquire the target network and the image to be processed, wherein the target network includes the updated feature extraction network and the downstream task network; The target network is used to process the image to be processed in order to obtain the processing result.

11. A model training device, characterized in that, The device includes: The acquisition module is used to acquire a sample image, which includes multiple objects; A sampling module is used to sample the sample image to obtain a first image block and a second image block, wherein the first image block and the second image block are different image blocks on the sample image; The feature extraction module is used to extract features from the first image block and the second image block respectively through a feature extraction network based on the fact that the intersection ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold, so as to obtain a first feature map and a second feature map, wherein the first feature map and the second feature map are two-dimensional feature maps; The model update module is used to determine the loss based on the difference between the first feature map and the second feature map, and update the feature extraction network based on the loss to obtain the updated feature extraction network.

12. The apparatus according to claim 11, characterized in that, The threshold is a value greater than or equal to 0.

4.

13. The apparatus according to claim 11 or 12, characterized in that, The device further includes: An alignment module is used to align the first image block and the second image block before determining the loss based on the difference between the first feature map and the second feature map, so as to obtain an aligned first image block and an aligned second image block. The model update module is specifically used for: The loss is determined based on the difference between the aligned first feature map and the aligned second feature map.

14. The apparatus according to claim 13, characterized in that, The sample image includes a target region, which is the overlapping area where the first image block and the second image block are located on the sample image. The alignment module is specifically used for: Based on the target region, a first sub-feature map corresponding to the target region in the first feature map and a second sub-feature map corresponding to the target region in the second feature map are determined. The first sub-feature map is upsampled to obtain the aligned first image block; The second sub-feature map is upsampled to obtain the aligned second image block, wherein the aligned first image block has the same size as the aligned second image block.

15. The apparatus according to claim 13, characterized in that, The first feature map and the second feature map have the same size. The first feature map includes M first feature points, and the second feature map includes M second feature points. The M first feature points correspond to M first pixels in the sample image, and the M second feature points correspond to M second pixels in the sample image. There is a one-to-one correspondence between the M first pixels and the M second pixels. The alignment module is specifically used for: A third feature map is obtained based on the M first pixels and the M second pixels. The third feature map and the second feature map have the same size. The third feature map includes M third feature points, each of which is obtained based on the pixel position difference between the corresponding first pixels and second pixels. The third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block.

16. The apparatus according to claim 15, characterized in that, The alignment module is specifically used for: The third feature map is stitched together with the first feature map in the depth direction.

17. The apparatus according to claim 11, characterized in that, The model update module is specifically used for: The target prediction network processes the M first feature points of the first feature map to obtain the predicted value of each first feature point. Based on the target clustering algorithm, the M second feature points of the first feature map are clustered to update the feature values ​​of the M second feature points, wherein the feature value of each updated second feature point is the cluster center feature value of the cluster category in which it belongs. The loss is determined based on the difference between the predicted value of each first feature point and the feature value of each updated second feature point.

18. The apparatus according to claim 17, characterized in that, The model update module is also used for: The target prediction network is updated based on the loss.

19. The apparatus according to claim 11, characterized in that, The objects include at least one of people, vehicles, traffic signs, lane lines, and plants.

20. The apparatus according to claim 11, characterized in that, The device further includes: The data processing module is used to acquire the target network and the image to be processed, wherein the target network includes the updated feature extraction network and the downstream task network; The target network is used to process the image to be processed in order to obtain the processing result.

21. A model training device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and perform the method as described in any one of claims 1 to 10.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that, The computer program product includes code that, when executed, performs the steps of the method according to any one of claims 1 to 10.

Images