Visual perception model training method, device, equipment and storage medium

Abstract

The present application discloses a visual perception model training method, apparatus, device and storage medium, which belongs to the field of computer vision technology. The method first obtains multiple single-instance sample images, and then performs a first data enhancement process and a second data enhancement process on each single-instance sample image to obtain a first processed image and a second processed image, and then performs a splicing process on M single-instance sample images selected from the multiple single-instance sample images to obtain a multi-instance sample image, and then performs feature extraction on the first processed image, the multi-instance sample image, the second processed image and the M single-instance sample images to obtain corresponding image features. Based on these image features, a model is trained to obtain a visual perception model. Since the image features of the training model include not only the image features of the single-instance sample image, but also the image features of the multi-instance sample image, the generalization ability of the model can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a visual perception model training method, apparatus, device and storage medium. Background Art

[0002] In the field of computer vision, visual perception tasks include image classification, object detection, and instance segmentation. Accordingly, visual perception models can be used for image classification, object detection, and instance segmentation. These models can be trained using either supervised or self-supervised learning. Self-supervised learning, which requires no data labels for model training, is gaining increasing popularity.

[0003] When training a visual perception model through self-supervised learning, the related technology only uses a single-instance sample image as sample data. After performing data enhancement processing and feature extraction on the single-instance sample image in sequence, the model is trained based on the extracted features to obtain a visual perception model.

[0004] Since the sample data used to train the visual perception model only includes single-instance sample images and the data is relatively simple, the trained visual perception model cannot perform visual perception tasks in complex scenes and has poor generalization ability. Summary of the Invention

[0005] The present invention provides a method, apparatus, device, and storage medium for training a visual perception model, which can improve the generalization capability of the model. The technical solution is as follows:

[0006] In one aspect, a method for training a visual perception model is provided, the method comprising:

[0007] Acquire a plurality of single-instance sample images, each of which includes an object;

[0008] For each single-instance sample image, performing a first data enhancement process and a second data enhancement process on the single-instance sample image to obtain a first processed image and a second processed image;

[0009] Selecting M single-instance sample images from the multiple single-instance sample images, and performing splicing processing on the M single-instance sample images to obtain a multi-instance sample image; wherein M is a positive integer greater than 1;

[0010] performing feature extraction on the first processed image, the multi-instance sample image, the second processed image, and the M single-instance sample images, respectively, to obtain a first image feature, a second image feature, a third image feature, and M fourth image features;

[0011] Model training is performed based on the first image feature, the second image feature, the third image feature and the M fourth image features to obtain a visual perception model.

[0012] In one aspect, a visual perception model training device is provided, comprising:

[0013] An acquisition module, configured to acquire a plurality of single-instance sample images, each of which includes an object;

[0014] A first processing module is configured to perform a first data enhancement process and a second data enhancement process on each single instance sample image, respectively, to obtain a first processed image and a second processed image;

[0015] a second processing module, configured to select M single-instance sample images from the plurality of single-instance sample images, and perform splicing processing on the M single-instance sample images to obtain a multi-instance sample image; wherein M is a positive integer greater than 1;

[0016] a feature extraction module, configured to perform feature extraction on the first processed image, the multi-instance sample image, the second processed image, and the M single-instance sample images, respectively, to obtain a first image feature, a second image feature, a third image feature, and M fourth image features;

[0017] A training module is used to perform model training based on the first image feature, the second image feature, the third image feature and the M fourth image features to obtain a visual perception model.

[0018] On the one hand, an electronic device is provided, which includes a processor and a memory, wherein at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor to implement the above-mentioned visual perception model training method.

[0019] On the one hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by the processor to implement the above-mentioned visual perception model training method.

[0020] On the one hand, a computer program product is provided, in which at least one program code is stored, and the at least one program code is loaded and executed by a processor to implement the above-mentioned visual perception model training method.

[0021] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0022] The embodiment of the present application provides a method for training a visual perception model. The method obtains a multi-instance sample image by splicing M single-instance sample images, and then trains the model using the image features of the multi-instance sample images and the image features of the single-instance sample images. It can be seen that the image features of the training model include not only the image features of the single-instance sample images, but also the image features of the multi-instance sample images. The image features are relatively rich. Therefore, the visual perception model trained by this method can not only perform visual perception tasks in simple scenes corresponding to single-instance sample images, but also perform visual perception tasks in complex scenes corresponding to multi-instance sample images, thereby improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 Schematic diagram of an implementation environment of a visual perception model training method provided in an embodiment of the present application;

[0025] Figure 2 This is a flowchart of a visual perception model training method provided by an embodiment of the present application;

[0026] Figure 3 This is a flowchart of a visual perception model training method provided by an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of a framework for training a visual perception model provided in an embodiment of the present application;

[0028] Figure 5 This is a schematic diagram of a method for extracting features from multiple sample images provided by an embodiment of the present application;

[0029] Figure 6 This is a schematic diagram comparing other models with the model provided in this application on the instance segmentation task;

[0030] Figure 7 This is a structural diagram of a visual perception model training device provided in an embodiment of the present application;

[0031] Figure 8 This is a structural block diagram of a terminal provided in an embodiment of the present application;

[0032] Figure 9 This is a structural block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0034] The terms "first," "second," "third," and "fourth," etc. in the specification and claims of this application and the accompanying drawings are used to distinguish different objects, not to describe a specific order. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0035] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample images, image features, and visual perception tasks involved in this application were all obtained with full authorization.

[0036] Figure 1 This is a schematic diagram of an implementation environment for a visual perception model training method provided in an embodiment of the present application. Figure 1 The implementation environment includes: an electronic device, which can be provided as a terminal 101, a server 102, or a terminal 101 and a server 102. If the electronic device is provided as a terminal 101 and a server 102, the terminal 101 and the server 102 can be connected via a wireless or wired network. In the embodiments of the present application, the electronic device is not specifically limited.

[0037] If the electronic device is provided as a terminal 101, the terminal 101 performs model training to obtain a visual perception model. The visual perception model can be deployed in the terminal 101, so that the terminal 101 can perform visual perception tasks using the visual perception model.

[0038] If the electronic device is provided as the server 102, the server 102 performs model training to obtain a visual perception model. The visual perception model can be deployed to the terminal 101. Accordingly, the implementation environment also includes: the terminal 101, which can perform visual perception tasks using the visual perception model.

[0039] If the electronic device is provided as a terminal 101 and a server 102, the server 102 performs model training to obtain a visual perception model, and then the visual perception model is deployed to the terminal 101, so that the terminal 101 can perform visual perception tasks through the visual perception model.

[0040] In an embodiment of the present application, the visual perception task includes but is not limited to image classification tasks, target detection tasks, and instance segmentation tasks. For example, the visual perception task is a lane line detection task, a road element recognition task, a face detection task, etc. Accordingly, the visual perception model can be widely used in business scenarios such as high-precision maps, autonomous driving, and road perception.

[0041] It should be noted that the visual perception model provided in the embodiments of the present application is a pre-trained visual perception model. Before being deployed in the terminal 101, the pre-trained visual perception model can be further trained based on sample images corresponding to the visual perception task to be performed to obtain a target visual perception model. The target visual perception model can be deployed in the terminal 101, so that the terminal 101 performs the visual perception task using the target visual perception model.

[0042] The terminal 101 is at least one of a mobile phone, a tablet computer, a PC (Personal Computer), an intelligent voice interaction device, and a vehicle-mounted terminal. The server 102 can be at least one of a single server, a server cluster consisting of multiple servers, a cloud server, a cloud computing platform, and a virtualization center.

[0043] Figure 2 This is a flowchart of a visual perception model training method provided in an embodiment of the present application, see Figure 2 , the method comprising:

[0044] Step 201: Acquire multiple single-instance sample images, each of which includes an object.

[0045] The object may be an object, a vehicle, an animal or a person, without specific limitation.

[0046] Step 202: For each single-instance sample image, perform a first data enhancement process and a second data enhancement process on the single-instance sample image to obtain a first processed image and a second processed image.

[0047] Data augmentation processing includes global augmentation and local cropping. Global augmentation mainly enhances the entire single-instance sample image and includes at least one of random cropping, random flipping, random noise, and color change. Local cropping mainly crops a part of the single-instance sample image.

[0048] Step 203: Select M single-instance sample images from the multiple single-instance sample images, and perform splicing processing on the M single-instance sample images to obtain a multi-instance sample image.

[0049] M is a positive integer greater than 1, for example, M is 2, 3, or 4. In the embodiment of the present application, only M is 4 as an example for description. Accordingly, if M is 4, the multi-instance sample image is obtained by splicing 4 single-instance sample images.

[0050] Step 204: performing feature extraction on the first processed image, the multi-instance sample image, the second processed image, and the M single-instance sample images respectively to obtain first image features, second image features, third image features, and M fourth image features.

[0051] In this step, feature extraction can be performed on the image using a residual network or other network. Here, feature extraction of the image using a residual network is used as an example. Accordingly, the residual network includes a first residual network and a second residual network. In this step, feature extraction is performed on the first processed image and the multi-instance sample images using the first residual network to obtain first image features and second image features. Feature extraction is performed on the second processed image and the M single-instance sample images using the second residual network to obtain third image features and M fourth image features.

[0052] Among them, the parameter updating method of the first residual network is different from the parameter updating method of the second residual network, and the parameter updating speed of the first residual network is greater than the parameter updating speed of the second residual network.

[0053] Step 205: Perform model training based on the first image feature, the second image feature, the third image feature and M fourth image features to obtain a visual perception model.

[0054] The embodiment of the present application provides a method for training a visual perception model. The method obtains a multi-instance sample image by splicing M single-instance sample images, and then trains the model using the image features of the multi-instance sample images and the image features of the single-instance sample images. It can be seen that the image features of the training model include not only the image features of the single-instance sample images, but also the image features of the multi-instance sample images. The image features are relatively rich. Therefore, the visual perception model trained by this method can not only perform visual perception tasks in simple scenes corresponding to single-instance sample images, but also perform visual perception tasks in complex scenes corresponding to multi-instance sample images, thereby improving the generalization ability of the model.

[0055] Figure 3 This is a flowchart of a visual perception model training method provided by an embodiment of the present application, which is executed by an electronic device. Figure 3 , the method comprising:

[0056] Step 301: The electronic device obtains a plurality of single-instance sample images.

[0057] A single-instance sample image includes an object, which can be an object, vehicle, animal, or person, without specific limitations. For example, when a visual perception model is applied in a high-precision map or road perception scenario, the object can be a road element such as a vehicle, streetlight, guardrail, traffic light, or lane marking.

[0058] The electronic device may obtain multiple single-instance sample images from the unlabeled dataset. The number of the single-instance sample images may be set and changed as needed, and is not specifically limited thereto.

[0059] In an embodiment of the present application, the electronic device may perform model training based on at least one of the global features and local features of a single-instance sample image, together with the image features of multiple-instance sample images. Accordingly, the process includes the following three implementation methods:

[0060] In a first implementation method, the electronic device performs model training based on the global features of a single-instance sample image and the image features of multiple-instance sample images.

[0061] In a second implementation method, the electronic device performs model training based on the local features of the single-instance sample image and the image features of the multiple-instance sample images.

[0062] In a third implementation method, the electronic device performs model training based on the global features of a single-instance sample image, the local features of a single-instance sample image, and the image features of multiple-instance sample images.

[0063] For the first implementation, model training based solely on the global features of a single-instance sample image, in addition to the image features of multiple-instance sample images, results in poor performance in fine-grained feature perception, such as object detection and instance segmentation. For the second implementation, model training based solely on the local features of a single-instance sample image, in addition to the image features of multiple-instance sample images, results in poor performance in global feature perception, such as classification tasks.

[0064] Therefore, in this embodiment of the present application, the electronic device performs model training based on the global features of the single-instance sample image, the local features of the single-instance sample image, and the image features of the multiple-instance sample images, which is the third implementation method mentioned above. Accordingly, after executing step 301, the electronic device executes step 302.

[0065] In an embodiment of the present application, the electronic device improves the model's ability to learn the main features of the image through the global features of a single-instance sample image, improves the model's ability to learn fine-grained features through the local features of a single-instance sample image, improves the accuracy of the model's instance boundary regression, and improves the model's ability to locate the centers of multiple instances in the image through the image features of multiple-instance sample images, thereby ultimately improving the model's performance in multiple visual perception tasks such as classification tasks, target detection tasks, and instance segmentation tasks.

[0066] Step 302: For each single instance sample image, the electronic device performs a first data enhancement process and a second data enhancement process on the single instance sample image to obtain a first processed image and a second processed image.

[0067] The first data enhancement processing includes: a first global enhancement processing and a first local cropping processing, and the second data enhancement processing includes: a second global enhancement processing and a second local cropping processing. Accordingly, the first processed image includes: a first global image and a first local image, and the second processed image includes: a second global image and a second local image.

[0068] This step can be achieved by following the steps (1) to (2), including:

[0069] (1) The electronic device performs a first global enhancement process and a first local cropping process on the single instance sample image to obtain a first global image and a first local image.

[0070] Global enhancement primarily involves enhancing the entire single-instance sample image, including at least one of random cropping, random flipping, random noise, and color change. Global enhancement can also include other processes, which are not listed here. Local cropping primarily involves randomly cropping a portion of a single-instance sample image.

[0071] Correspondingly, the electronic device performs a first global enhancement process on the single instance sample image through a first data enhancement operator to obtain a first global image, and performs a first local cropping process on the single instance sample image through a first image cropping operator to obtain a first local image, see Figure 4 .

[0072] (2) The electronic device performs a second global enhancement process and a second local cropping process on the single instance sample image to obtain a second global image and a second local image.

[0073] The electronic device can perform a second global enhancement process on the single instance sample image through a second data enhancement operator to obtain a second global image, and perform a second local cropping process on the single instance sample image through a second image cropping operator to obtain a second local image, wherein there is an overlapping area between the first local image and the second local image.

[0074] For the first partial image and the second partial image, the electronic device performs the first partial cropping process and the second partial cropping process on the single instance sample image, respectively, to obtain the fifth partial image and the sixth partial image. The electronic device determines whether there is an overlapping area between the fifth partial image and the sixth partial image. If there is an overlapping area between the fifth partial image and the sixth partial image, the electronic device determines the image of the overlapping area in the fifth partial image as the first partial image, and determines the image of the overlapping area in the sixth partial image as the second partial image. If there is no overlapping area between the fifth partial image and the sixth partial image, the electronic device re-performs the first partial cropping process and the second partial cropping process on the single instance sample image, respectively, until there is an overlapping area between the two partial images obtained after cropping, thereby obtaining the first partial image and the second partial image.

[0075] It should be noted that the first and second global enhancement processes are different. For example, the first global enhancement process may be a color change, while the second global enhancement process may be a random flip. Alternatively, the first global enhancement process may be to change the color of a single-instance sample image from its current color to a first color, while the second global enhancement process may be to change the color of the single-instance sample image from its current color to a second color.

[0076] In an embodiment of the present application, the electronic device may simultaneously execute step (1) and step (2), or the electronic device may first execute step (1) and then execute step (2), or the electronic device may first execute step (2) and then execute step (1), or the electronic device may first simultaneously execute the first global enhancement processing of the single instance sample image in step (1) and the second global enhancement processing of the single instance sample image in step (2), and then simultaneously execute the first local cropping processing of the single instance sample image in step (1) and the second local cropping processing of the single instance sample image in step (2). In the embodiment of the present application, there is no specific limitation on this execution order.

[0077] In an embodiment of the present application, the electronic device performs different global enhancements on a single-instance sample image to obtain different global images. Subsequently, the electronic device can learn single-instance global features based on these different global images, thereby improving the model's performance on classification tasks. Furthermore, by performing different local cropping processes on a single-instance sample image to obtain different local images, the electronic device can learn single-instance local features based on these different local images, thereby improving the model's performance on fine-grained tasks such as object detection and instance segmentation.

[0078] Step 303: the electronic device selects M single-instance sample images from the multiple single-instance sample images, and performs splicing processing on the M single-instance sample images to obtain a multi-instance sample image.

[0079] This step can be achieved by following the steps (1) to (3), including:

[0080] (1) The electronic device selects M single-instance sample images from multiple single-instance sample images.

[0081] In the embodiment of the present application, M is a positive integer greater than 1, for example, M is 2, 3, or 4. In the embodiment of the present application, only the case where M is 4 is used as an example for description. Accordingly, if M is 4, step (1) can be replaced by: the electronic device selects 4 single-instance sample images from the multiple single-instance sample images.

[0082] The electronic device may randomly select M single instance sample images from a plurality of single instance sample images, or may select M single instance sample images in a certain order, which is not specifically limited.

[0083] (2) The electronic device performs a third global enhancement process on each of the M single instance sample images to obtain M third processed images.

[0084] For each single-instance sample image among the M single-instance sample images, the process by which the electronic device performs the third global enhancement processing on the single-instance sample image is similar to the process by which the electronic device performs the first global enhancement processing or the second global enhancement processing on the single-instance sample image in step 302, and will not be repeated here.

[0085] If M is 4, step (2) can be replaced by: the electronic device performs the third global enhancement processing on the four single-instance sample images respectively to obtain four third processed images.

[0086] (3) The electronic device performs splicing processing on the M third processed images to obtain multiple instance sample images.

[0087] The electronic device splices the M third processed images to obtain multiple instance sample images, which are one image.

[0088] If M is 4, step (3) can be replaced by: the electronic device performs splicing processing on the four third processed images to obtain multiple instance sample images.

[0089] In this implementation, the electronic device can splice the four third processed images in the order of upper left, upper right, lower left and lower right to obtain multiple instance sample images. Figure 4 .

[0090] In an embodiment of the present application, generating a multi-instance sample image from multiple single-instance sample images helps improve the model's ability to locate the centers of multiple instances in the image, enabling the model to perform visual perception tasks in complex scenes, thereby improving the model's generalization ability.

[0091] Step 304: The electronic device performs feature extraction on the first processed image and the multiple instance sample images respectively to obtain first image features and second image features.

[0092] This step can be achieved by following the steps (1) to (2), including:

[0093] (1) The electronic device extracts features from the first processed image to obtain first image features.

[0094] According to step 302, the first processed image includes: a first global image and a first local image, and the first image feature includes: a first global feature of the first global image and a first local feature of the first local image. Then, in step (1), the electronic device performs feature extraction on the first global image and the first local image respectively to obtain the first global feature and the first local feature.

[0095] In this implementation, the electronic device extracts features from the first global image and the first local image respectively through the first residual network to obtain a first global feature and a first local feature. The first local feature includes N first sub-features. Figure 4 .

[0096] The electronic device extracts features from the first partial image using the first residual network to obtain the first partial features. The process can be implemented by following steps (A-1) to (A-4), including:

[0097] (A-1): The electronic device extracts features from the first local image through a first residual network to obtain second local features.

[0098] In view of the generalized use of residual networks, the backbone network in the embodiment of the present application uses residual networks to extract features from images. Of course, electronic devices can also extract features from images through other networks, and the feature extraction method is not specifically limited here.

[0099] Among them, the parameter update method of the first residual network can be set and changed as needed. Here, the parameter update method of the first residual network is only described as back propagation to gradient update the parameters as an example.

[0100] (A-2) The electronic device adjusts the image size of the first partial image to obtain a third partial image.

[0101] Since the first partial image is a randomly cropped image, the image size of the first partial image does not match the preset image size. The electronic device adjusts the image size of the first partial image based on the preset image size to obtain a third partial image, and the image size of the third partial image is equal to the preset image size.

[0102] One thing that needs to be explained is that the electronic device can adjust the image size of the first local image after obtaining the first local image in step 302. Then, in step 304, after executing the second local feature, the electronic device directly executes step (A-3) without executing step (A-2). If the electronic device does not adjust the image size of the first local image after obtaining the first local image in step 302, the electronic device executes step (A-2). Among them, the electronic device can first execute step (A-1) and then execute step (A-2), or the electronic device can first execute step (A-2) and then execute step (A-1). There is no specific limitation on this execution order. For the first global image, the second global image and the multi-instance sample image, the electronic device also needs to adjust the image size of the image before performing feature extraction.

[0103] (A-3) The electronic device performs a region pooling process on the second local feature based on multiple pixel coordinates of the third partial image to obtain N second sub-features.

[0104] After the electronic device obtains the third partial image, it can determine multiple pixel coordinates in the third partial image, and based on the multiple pixel coordinates in the third partial image, perform regional pooling processing on the third partial image through a region of interest pooling operator to generate N second sub-features corresponding to N regions, with one region corresponding to one second sub-feature.

[0105] The electronic device may determine the pixel coordinates of the upper left corner and the lower right corner of the third partial image, and divide the third partial image into N areas based on the pixel coordinates of the upper left corner and the lower right corner, where N is a positive integer greater than 1.

[0106] The size of N can be set and changed as needed and is not specifically limited thereto. For example, N is 4, 6, or 9. In the embodiment of the present application, only N of 9 is used as an example for description. Accordingly, if N is 9, then in step (A-3), the electronic device obtains 9 second sub-features.

[0107] In addition, the size of each of the N regions may be the same or different. In the embodiment of the present application, only the example of the case where the size of each of the N regions is the same is used for illustration.

[0108] (A-4) The electronic device sequentially performs a first convolution process, a first correction process, and a second convolution process on the N second sub-features to obtain N first sub-features.

[0109] The electronic device may first perform a first convolution process on the N second sub-features through continuous 1×1 convolutional layers, then perform a first correction process on the N second sub-features after the first convolution process through a rectified linear unit, then perform a second convolution process on the N second sub-features after the first correction process through a 1×1 convolutional layer, and finally perform feature flattening on the N second sub-features after the second convolution process to obtain N first sub-features, where the N first sub-features are first sub-features with 1 row and N columns. The correction process mainly refers to adjusting the size of the features.

[0110] For example, if N is 9, the electronic device in step (A-3) divides the third local image into 3×3 areas, and the obtained 9 second sub-features are 3 rows and 3 columns of second sub-features. The electronic device flattens the 9 second sub-features after the second convolution processing to change the 3 rows and 3 columns of second sub-features into 1 row and 9 columns of second sub-features, which facilitates subsequent pixel-level comparative learning.

[0111] In an embodiment of the present application, a constraint mechanism for local image generation is designed to ensure the adequacy of local feature learning, which helps to improve the performance of the model on fine-grained tasks.

[0112] In an embodiment of the present application, the electronic device performs feature extraction on the first global image through the first residual network to obtain the first global feature. The process is: the electronic device performs feature extraction on the first global image through the first residual network to obtain a third global feature, performs average pooling processing on the third global feature to obtain a fourth global feature, and performs third correction processing and full connection processing on the fourth global feature in sequence to obtain the first global feature.

[0113] In this implementation, the electronic device performs average pooling processing on the third global feature, where the average pooling processing is performed on the entire global image to obtain the fourth global feature, and then performs the third correction processing and the full connection processing on the fourth global feature in sequence to obtain the first global feature.

[0114] (2) The electronic device extracts features from the multiple instance sample images to obtain second image features.

[0115] The electronic device extracts features from multiple instance sample images through a first residual network to obtain a fifth image feature, performs average pooling processing on the fifth image feature to obtain M third sub-features, and sequentially performs a second correction processing and a full connection processing on the M third sub-features to obtain a second image feature.

[0116] In this implementation, the electronic device extracts features from the multi-instance sample images using a first residual network to obtain a fifth image feature. The fifth image feature can be represented as a dimensional feature of (n, 2048, H / 32, W / 32), where n represents the amount of data in a batch, 2048 represents the channel dimension of the feature, H and W represent the height and width of the multi-instance sample image, respectively, and 32 represents the image bit depth. Since the multi-instance sample image is obtained by splicing M single-instance sample images, the multi-instance sample image includes M single-instance sample images. The electronic device can then perform average pooling processing on the fifth image feature based on the M single-instance sample images to obtain M third sub-features, and then perform a second correction process and a full-connection process on the M third sub-features in sequence to obtain a second image feature.

[0117] See also Figure 5 If M is 4, then after performing average pooling on the n fifth image features, n*4 third sub-features are obtained. Then, after performing the second correction process and full connection process on the n*4 third sub-features, n*4 sub-features are obtained, and the dimension of each sub-feature is 128. Among them, one second image feature corresponds to 4 sub-features.

[0118] In an embodiment of the present application, extracting image features of multi-instance sample images through the above method helps the model to fully learn multi-instance features, enabling the model to perform visual perception tasks in complex scenes, thereby improving the generalization ability of the model.

[0119] Step 305: The electronic device performs feature extraction on the second processed image and the M single-instance sample images respectively to obtain a third image feature and M fourth image features.

[0120] This step can be achieved by following the steps (1) to (2), including:

[0121] (1) The electronic device extracts features from the second processed image to obtain third image features.

[0122] According to step 302, the second processed image includes: a second global image and a second local image, and the third image feature includes: a second global feature of the second global image and a second local feature of the second local image. In step (1), the electronic device extracts features from the second global image and the second local image, respectively, to obtain a second global feature and a second local feature.

[0123] In this implementation, the electronic device performs feature extraction on the second global image and the second local image respectively through the second residual network to obtain second global features and second local features, where the second local features include N fourth sub-features.

[0124] The electronic device performs feature extraction on the second global image through the second residual network to obtain the second global feature, which is similar to the process in which the electronic device performs feature extraction on the first global image through the first residual network to obtain the first global feature. The electronic device performs feature extraction on the second local image through the second residual network to obtain the second local feature, which is similar to the process in which the electronic device performs feature extraction on the first local image through the first residual network to obtain the first local feature, and will not be repeated here.

[0125] It should be noted that the parameter update method of the first residual network is different from that of the second residual network, and the parameter update speed of the first residual network is faster than that of the second residual network. For example, if the parameter update method of the first residual network is gradient update using backpropagation, the parameter update speed is represented by V1, and the parameter update method of the second residual network is momentum update, the parameter update speed is represented by V2, then V2 = r V1. Where r is a coefficient, 0 < r < 1.

[0126] In the embodiment of the present application, image features are extracted by a second residual network with a smaller parameter update speed, which can avoid excessively fast feature updates, resulting in unstable model training and failure to converge.

[0127] (2) The electronic device performs feature extraction on the M single instance sample images to obtain M fourth image features.

[0128] For each of the M single-instance sample images, the electronic device extracts features of the single-instance sample image through the second residual network to obtain a fourth image feature. Figure 4 .

[0129] The process in which the electronic device extracts features from the single instance sample image through the second residual network to obtain the fourth image feature is similar to the process in which the electronic device extracts features from the second global image through the second residual network to obtain the second global feature, and will not be repeated here.

[0130] Step 306: The electronic device constructs a first loss function based on the first global feature and the second global feature.

[0131] This step can be achieved by following the steps (1) to (4), including:

[0132] (1) The electronic device constructs the third positive sample pair.

[0133] The third positive sample pair includes a first global feature and a second global feature obtained from the same single-instance sample image.

[0134] For each single instance sample image, the electronic device forms a third positive sample pair with the first global feature and the second global feature corresponding to the single instance sample image.

[0135] (2) The electronic device constructs a third negative sample pair.

[0136] The third negative sample pair includes a first global feature and a second global feature obtained from different single-instance sample images, and the first global feature in the third negative sample pair is the same feature as the first global feature in the third positive sample pair.

[0137] For each single-instance sample image, the electronic device forms a third negative sample pair with the first global feature corresponding to the single-instance sample image and the second global feature corresponding to other single-instance sample images.

[0138] For example, the electronic device combines the first global feature corresponding to the first single instance sample image and the second global feature into a third positive sample pair, and combines the first global feature corresponding to the first single instance sample image and the second global feature corresponding to the second single instance sample image into a third negative sample pair.

[0139] For each single-instance sample image, the electronic device can pre-store the second global features corresponding to other single-instance sample images except the single-instance sample image in the first feature dictionary, and dynamically update the first feature dictionary as the features are updated. When constructing the third negative sample pair, each second global feature in the first feature dictionary is respectively combined with the first global feature of the single-instance sample image to form a third negative sample pair, thereby improving the efficiency of constructing the third negative sample pair.

[0140] The number of second global features stored in the first feature dictionary can be set and changed as needed. For example, the number is 65536, which means that for each single-instance sample image, the number of corresponding third negative sample pairs is 65536.

[0141] (3) The electronic device determines the sixth similarity and the seventh similarity.

[0142] For each third positive sample pair, the sixth similarity is used to measure the similarity between the first global feature and the second global feature in the third positive sample pair. Accordingly, the electronic device determines the similarity between the first global feature and the second global feature in the third positive sample pair to obtain the sixth similarity.

[0143] For each third negative sample pair, the seventh similarity is used to measure the similarity between the first global feature and the second global feature in the third negative sample pair. Accordingly, the electronic device determines the similarity between the first global feature and the second global feature in the third negative sample pair to obtain the seventh similarity.

[0144] Among them, the electronic device can determine their similarity by determining the distance between the first global feature and the second global feature in the third positive sample pair. Similarly, the electronic device can also determine their similarity by determining the distance between the first global feature and the second global feature in the third negative sample pair.

[0145] (4) The electronic device constructs a first loss function based on the sixth similarity and the seventh similarity.

[0146] The electronic device may construct a first loss function based on the sixth similarity and the seventh similarity using the following formula:

[0147]

[0148] Where L1 represents the first loss function, q represents the first global feature in the third positive sample pair or the first global feature in the third negative sample pair, and k + represents the second global feature in the third positive sample pair, k - represents the second global feature in the third negative sample pair, τ1 is the coefficient, 0<τ1<1, exp(q·k + / τ1) represents the sixth similarity, exp(q·k - / τ1) represents the seventh similarity.

[0149] According to step (2), for each single instance sample image, there are multiple third negative sample pairs corresponding to the single instance sample image. Therefore, Represents the sum of the seventh similarities corresponding to multiple third-negative sample pairs. For example, for each single-instance sample image, the number of its corresponding third-negative sample pairs is 65536, then It represents the sum of the seventh similarities corresponding to 65536 third negative sample pairs.

[0150] Step 307: The electronic device constructs a second loss function based on the first local feature and the second local feature.

[0151] This step can be achieved by following the steps (1) to (4), including:

[0152] (1) The electronic device constructs a second positive sample pair.

[0153] The second positive sample pair includes N first sub-features and N fourth sub-features, and the N first sub-features and the N fourth sub-features correspond to the same single instance sample image.

[0154] For each single-instance sample image, the electronic device combines the first local feature and the second local feature corresponding to the single-instance sample image into a second positive sample pair. Since the first local feature includes N first sub-features and the second local feature includes N fourth sub-features, the electronic device combines the N first sub-features and the N fourth sub-features corresponding to the single-instance sample image into a second positive sample pair.

[0155] (2) The electronic device constructs a second negative sample pair.

[0156] The second negative sample pair includes N first sub-features and a third local feature. The N first sub-features in the second negative sample pair are the same features as the N first sub-features in the second positive sample pair. The third local feature is obtained by performing feature extraction on a fourth local image. The fourth local image and the first local image are derived from the same single-instance sample image, but the fourth local image is an image that has no overlapping area with the first local image.

[0157] For each single instance sample image, the electronic device combines the first local feature corresponding to the single instance sample image and the third local feature corresponding to the single instance sample image into a second negative sample pair.

[0158] The electronic device may also pre-store the third local feature corresponding to the single instance sample image in a second feature dictionary, and dynamically update the second feature dictionary, and when constructing a second negative sample pair, each third local feature in the second feature dictionary is respectively combined with the N first sub-features to form a second negative sample pair.

[0159] The number of third local features stored in the second feature dictionary can be set and changed as needed. For example, the number is 64512, which means that for each single-instance sample image, the number of corresponding second negative sample pairs is 64512.

[0160] (3) The electronic device determines the fourth similarity and the fifth similarity.

[0161] For each second positive sample pair, the fourth similarity is used to measure the similarity between the N first sub-features and the N fourth sub-features. Accordingly, for each first sub-feature, the electronic device determines the similarity between the first sub-feature and its corresponding fourth sub-feature to obtain an eighth similarity, and determines the fourth similarity based on the N eighth similarities.

[0162] For each second negative sample pair, the fifth similarity is used to measure the similarity between the N first sub-features and the third local feature. Accordingly, for each first sub-feature, the electronic device determines the similarity between the first sub-feature and each third local feature to obtain multiple ninth similarities. Based on the multiple ninth similarities corresponding to each first sub-feature, the fifth similarity corresponding to the N first sub-features is determined.

[0163] (4) The electronic device constructs a second loss function based on the fourth similarity and the fifth similarity.

[0164] The electronic device may construct a second loss function based on the fourth similarity and the fifth similarity using the following formula:

[0165]

[0166] Among them, L2 represents the second loss function, q i represents the first sub-feature in the second positive sample pair or the first sub-feature in the second negative sample pair, 1≤i≤N, k i+ Represents the fourth sub-feature in the second positive sample pair, k′ - represents the third local feature in the second negative sample pair, τ2 is the coefficient, 0<τ2<1, exp(q i ·k i+ / τ2) represents the eighth similarity, Indicates the fourth similarity, exp(q i k′ - / τ2) represents the ninth similarity, Indicates the fifth similarity.

[0167] If N is 9, the second loss function can be expressed as:

[0168]

[0169] Step 308: The electronic device constructs a third loss function based on the second image feature and the M fourth image features.

[0170] This step can be achieved by following the steps (1) to (4), including:

[0171] (1) The electronic device constructs a first positive sample pair.

[0172] The first positive sample pair includes a second image feature and M fourth image features, where the second image feature is an image feature corresponding to a multi-instance sample image, and the multi-instance sample image is spliced ​​together from M single-instance sample images. The M single-instance sample images that constitute the multi-instance sample image are the same as the M single-instance sample images corresponding to the M fourth image features.

[0173] For each first positive sample pair, the electronic device forms a first positive sample pair with the second image features obtained from the M single-instance sample images and the M fourth image features corresponding to the M single-instance sample images.

[0174] (2) The electronic device constructs a first negative sample pair.

[0175] The first negative sample pair includes the second image feature and the second global feature. The M single-instance sample images corresponding to the second image feature do not include the single-instance sample image corresponding to the second global feature, that is, the single-instance sample image corresponding to the second global feature is an image other than the M single-instance sample images.

[0176] For each first negative sample pair, the electronic device combines the second image features corresponding to the multi-instance sample images and the second global features corresponding to the single-instance sample images other than the M single-instance sample images into a first negative sample pair.

[0177] In one possible implementation, since both the third negative sample pair and the first negative sample pair include the second global feature, and the second global feature is stored in the first feature dictionary, when constructing the first negative sample pair, the electronic device can obtain the second global feature corresponding to the single-instance sample image other than the M single-instance sample images from the first feature dictionary, and form the second global feature and the second image feature into a first negative sample pair.

[0178] In another possible implementation, for each single-instance sample image, the second global feature in the first feature dictionary corresponding to the single-instance sample image is the global feature corresponding to other single-instance sample images. For M single-instance sample images, the M single-instance sample images are composed of the single-instance sample image and (M-1) single-instance sample images. Therefore, the electronic device can delete the second local features corresponding to the (M-1) single-instance sample images from the first feature dictionary corresponding to the single-instance sample image to obtain a third feature dictionary. When constructing the first negative sample pair, the electronic device can also obtain the second global feature from the third feature dictionary and form the first negative sample pair with the second global feature and the second image feature.

[0179] From the two aforementioned implementations, it can be seen that the electronic device can obtain the second global feature from the first feature dictionary and form the second global feature and the second image feature into a first negative sample pair. The electronic device can also generate a third feature dictionary based on the first feature dictionary, obtain the second global feature from the third feature dictionary, and form the second global feature and the second image feature into a first negative sample pair. In the embodiments of the present application, the manner in which the electronic device obtains the second global feature is not specifically limited.

[0180] (3) The electronic device determines the first similarity and the second similarity.

[0181] The first similarity is used to measure the similarity between the second image feature and the M fourth image features. Accordingly, for each fourth image feature, the electronic device determines a third similarity between the second image feature and the fourth image feature, obtaining M third similarities, and determines the sum of the M third similarities as the first similarity.

[0182] The second similarity is used to measure the similarity between the second image feature and the second global feature. Accordingly, the electronic device determines the similarity between the second image feature and the second global feature to obtain the second similarity.

[0183] (4) The electronic device constructs a third loss function based on the first similarity and the second similarity.

[0184] The electronic device may construct a third loss function based on the first similarity and the second similarity using the following formula:

[0185]

[0186] Among them, L3 represents the third loss function, q m represents the second image feature in the first positive sample pair or the second image feature in the first negative sample pair, 1≤i′≤M,k i′+ represents the fourth image feature in the first positive sample pair, h - represents the second global feature in the first negative sample pair, τ3 is the coefficient, 0<τ3<1, exp(q m ·k i′+ / τ3) represents the third similarity, Indicates the first similarity, exp(q m ·h - / τ3) represents the second similarity.

[0187] If M is 4, the third loss function can be expressed as:

[0188]

[0189] Step 309: The electronic device performs model training based on the first loss function, the second loss function, and the third loss function to obtain a visual perception model.

[0190] The electronic device can sum the first loss function, the second loss function and the third loss function to obtain a total loss function, and perform model training based on the total loss function to obtain a visual perception model.

[0191] The electronic device may directly sum the first loss function, the second loss function, and the third loss function, or may perform a weighted sum of the first loss function, the second loss function, and the third loss function based on a first weight, a second weight, and a third weight, wherein the first weight is the weight corresponding to the first loss function, the second weight is the weight of the second loss function, and the third weight is the weight of the third loss function.

[0192] The electronic device performs model training based on the total loss function until the number of training iterations reaches the maximum number or the total loss value corresponding to the total loss function is within a preset range. The electronic device determines that the convergence condition is met and obtains a visual perception model.

[0193] In the embodiments of the present application, the electronic device performs joint learning based on the global and local features corresponding to a single-instance sample image and the image features corresponding to multiple-instance sample images. This learning process is end-to-end self-supervised learning, which improves the model's performance in classification, detection, and segmentation tasks. Furthermore, the learning of local features further improves the model's performance on fine-grained tasks.

[0194] The embodiment of the present application provides a method for training a visual perception model. The method obtains a multi-instance sample image by splicing M single-instance sample images, and then trains the model using the image features of the multi-instance sample images and the image features of the single-instance sample images. It can be seen that the image features of the training model include not only the image features of the single-instance sample images, but also the image features of the multi-instance sample images. The image features are relatively rich. Therefore, the visual perception model trained by this method can not only perform visual perception tasks in simple scenes corresponding to single-instance sample images, but also perform visual perception tasks in complex scenes corresponding to multi-instance sample images, thereby improving the generalization ability of the model.

[0195] In an embodiment of the present application, the visual perception model obtained by training through the above method is a pre-trained visual perception model. The visual perception model is further trained by sample images with labels to obtain the final target visual perception model.

[0196] The electronic device can determine sample images for further training the visual perception model based on the application scenario of the target visual perception model. For example, if the target visual perception model is applied to image classification, the sample images are images of different categories. If the target visual perception model is applied to face detection, the sample images are face images that include one or more faces. If the target visual perception model is applied to instance segmentation, the sample images are single-instance images or multi-instance images. If the target visual perception model is applied to road feature recognition, the sample images are images of road features such as vehicles, streetlights, guardrails, and lane markings.

[0197] When the target visual perception model is actually applied, the electronic device inputs the captured image or the image obtained from other devices into the target visual perception model, and the target visual perception model can be used to classify, detect or segment the image, and finally output the result.

[0198] In order to fully demonstrate that the model obtained in this application has indeed achieved a certain degree of improvement compared with other models, the following is an introduction to the performance comparison of the visual perception model obtained in this application and other models in classification tasks, see Table 1.

[0199] Table 1

[0200] Model Classification accuracy (%) MoCo 60.6 SeLa 61.5 PIRL 63.6 CPC v2 63.8 PCL 65.9 SimCLR v1 66.6 MoCo v2 67.5 This application model 70.3

[0201] It can be seen from Table 1 that compared with other models, the model obtained in this application has a higher classification accuracy, indicating that the model has improved performance in classification tasks.

[0202] The following is a comparison of the performance of the visual perception model obtained in this application and other models in object detection and instance segmentation tasks, see Table 2.

[0203] Table 2

[0204]

[0205] Among them, Mask R-CNN R-50FPN means that the network structure is Mask R-CNN, and its backbone network is the residual network ResNet50 with a feature pyramid structure, AP bb represents the target detection accuracy, It represents the average accuracy of target detection when the overlap threshold is 0.5. Indicates the average precision of target detection when the overlap threshold is 0.75, AP mk represents the average instance segmentation precision, represents the average precision of instance segmentation when the overlap threshold is 0.5, It represents the average precision of instance segmentation when the overlap threshold is 0.75.

[0206] It can be seen from Table 2 that compared with other models, the method of this application improves the accuracy of the model in fine-grained tasks such as target detection and instance segmentation, which improves the performance of the model in target detection tasks and instance segmentation tasks.

[0207] The above is to illustrate the performance of the model obtained by this application through data comparison. The following is to further illustrate the performance of the model obtained by this application through actual instance segmentation tasks. Figure 6 ,from Figure 6As can be seen in the figure: for the leftmost image in the first row, the image includes two instances. The model obtained by this application can segment both instances in the image, while the MoCo v2 model obtained in the related art can only segment one instance, see the white part in the figure. For the fourth image in the first row, the image includes three instances. The model obtained by this application can segment all three instances in the image, while the MoCo v2 model obtained in the related art can only segment one instance.

[0208] For the leftmost image in the second row, the image includes multiple instances. The model obtained by the present application can segment out multiple instances in the image, while the MoCo v2 model obtained in the related art can only segment out some instances and there are cases of mis-segmentation. For example, the MoCo v2 model mistakenly segments non-instance parts as instances, see the top white box in the third image in the second row, while the model obtained by the present application does not have this situation, see the second image in the second row. For the fourth image in the second row, the image includes an instance. The model obtained by the present application can segment out all instances in the image, while the MoCo v2 model obtained in the related art can also segment out the instance, but there are cases of mis-segmentation during the segmentation.

[0209] It can be seen that the model obtained in this application can not only accurately segment instances, but also avoid the influence of irrelevant factors and reduce the misrecognition rate.

[0210] In summary, the model obtained by the method of this application has not only been improved in classification tasks, but also in fine-grained tasks such as target detection and instance segmentation, thereby narrowing the gap between the backbone network in classification tasks and fine-grained tasks.

[0211] Figure 7 This is a structural diagram of a visual perception model training device provided in an embodiment of the present application, see Figure 7 , the device comprises:

[0212] An acquisition module 701 is configured to acquire a plurality of single-instance sample images, each of which includes an object;

[0213] A first processing module 702 is configured to perform a first data enhancement process and a second data enhancement process on each single-instance sample image, respectively, to obtain a first processed image and a second processed image;

[0214] The second processing module 703 is configured to select M single-instance sample images from the multiple single-instance sample images, and perform splicing processing on the M single-instance sample images to obtain a multi-instance sample image; wherein M is a positive integer greater than 1;

[0215] A feature extraction module 704 is configured to perform feature extraction on the first processed image, the multi-instance sample image, the second processed image, and the M single-instance sample images, respectively, to obtain a first image feature, a second image feature, a third image feature, and M fourth image features;

[0216] The training module 705 is used to perform model training based on the first image feature, the second image feature, the third image feature and M fourth image features to obtain a visual perception model.

[0217] In a possible implementation, the first data enhancement process includes: a first global enhancement process and a first local cropping process; the second data enhancement process includes: a second global enhancement process and a second local cropping process; the first processed image includes: a first global image and a first local image, and the second processed image includes: a second global image and a second local image;

[0218] The first processing module 702 is used to perform a first global enhancement process and a first local cropping process on the single-instance sample image to obtain a first global image and a first local image; and perform a second global enhancement process and a second local cropping process on the single-instance sample image to obtain a second global image and a second local image; wherein the first local image and the second local image have an overlapping area.

[0219] In another possible implementation, the second processing module 703 is configured to perform the third global enhancement processing on each of the M single-instance sample images to obtain M third processed images; and perform splicing processing on the M third processed images to obtain a multi-instance sample image.

[0220] In another possible implementation, the first processed image includes: a first global image and a first local image; the first image feature includes: a first global feature of the first global image and a first local feature of the first local image; the first local feature includes: N first sub-features; where N is a positive integer;

[0221] The feature extraction module 704 is used to extract features from the first global image through the first residual network to obtain a first global feature; extract features from the first local image through the first residual network to obtain a second local feature; adjust the image size of the first local image to obtain a third local image; perform regional pooling processing on the second local feature based on multiple pixel coordinates of the third local image to obtain N second sub-features; and perform a first convolution processing, a first correction processing, and a second convolution processing on the N second sub-features in sequence to obtain N first sub-features.

[0222] In another possible implementation, the feature extraction module 704 is used to perform feature extraction on multiple instance sample images through a first residual network to obtain a fifth image feature; perform average pooling processing on the fifth image feature to obtain M third sub-features; and perform a second correction processing and a full connection processing on the M third sub-features in sequence to obtain a second image feature.

[0223] In another possible implementation, the first image feature includes: a first global feature and a first local feature, and the third image feature includes: a second global feature and a second local feature;

[0224] Training module 705 is used to construct a first loss function based on the first global feature and the second global feature; construct a second loss function based on the first local feature and the second local feature; construct a third loss function based on the second image feature and M fourth image features; and perform model training based on the first loss function, the second loss function and the third loss function to obtain a visual perception model.

[0225] In another possible implementation, the training module 705 is used to construct a first positive sample pair, the first positive sample pair including the second image feature and M fourth image features; construct a first negative sample pair, the first negative sample pair including the second image feature and the second global feature; determine a first similarity and a second similarity; wherein the first similarity is used to measure the similarity between the second image feature and the M fourth image features, and the second similarity is used to measure the similarity between the second image feature and the second global feature; and construct a third loss function based on the first similarity and the second similarity.

[0226] In another possible implementation, the training module 705 is configured to determine, for each fourth image feature, a third similarity between the second image feature and the fourth image feature to obtain M third similarities; and determine a sum of the M third similarities as the first similarity.

[0227] In another possible implementation, the first local feature includes: N first sub-features, and the second local feature includes: N fourth sub-features;

[0228] Training module 705 is used to construct a second positive sample pair, the second positive sample pair includes N first sub-features and N fourth sub-features; construct a second negative sample pair, the second negative sample pair includes N first sub-features and a third local feature, the third local feature is obtained by feature extraction of a fourth local image, and the fourth local image is an image that has no overlapping area with the first local image; determine a fourth similarity and a fifth similarity; wherein the fourth similarity is used to measure the similarity between the N first sub-features and the N fourth sub-features, and the fifth similarity is used to measure the similarity between the N first sub-features and the third local feature; and construct a second loss function based on the fourth similarity and the fifth similarity.

[0229] An embodiment of the present application provides a visual perception model training device, which obtains a multi-instance sample image by splicing M single-instance sample images, and then performs model training based on the image features of the multi-instance sample images and the image features of the single-instance sample images. As can be seen, the image features of the training model include not only the image features of the single-instance sample images, but also the image features of the multi-instance sample images, and the image features are relatively rich. Therefore, the visual perception model trained by this device can not only perform visual perception tasks in simple scenes corresponding to single-instance sample images, but also perform visual perception tasks in complex scenes corresponding to multi-instance sample images, thereby improving the generalization ability of the model.

[0230] It should be noted that the visual perception model training device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when training the visual perception model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the electronic device can be divided into different functional modules to complete all or part of the functions described above. In addition, the visual perception model training device provided in the above embodiment and the visual perception model training method embodiment are of the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0231] If the electronic device is provided as a terminal, please refer to Figure 8 , Figure 8The following is a block diagram of a terminal 800 according to an exemplary embodiment of the present application. Terminal 800 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 800 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0232] Typically, the terminal 800 includes a processor 801 and a memory 802 .

[0233] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0234] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one program code instruction, which is used to be executed by the processor 801 to implement the visual perception model training method provided in the method embodiment of the present application.

[0235] In some embodiments, terminal 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 803 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 804, a touchscreen display 805, a camera 806, an audio circuit 807, a positioning component 808, and a power supply 809.

[0236] The peripheral device interface 803 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0237] The RF circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 804 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 804 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0238] Display screen 805 is used to display a user interface (UI). This UI can include graphics, text, icons, videos, or any combination thereof. When display screen 805 is a touchscreen display, it can also capture touch signals on or above the surface of display screen 805. These touch signals can be input as control signals to processor 801 for processing. Display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 805, located on the front panel of terminal 800. In other embodiments, there can be at least two display screens 805, located on different surfaces of terminal 800 or in a foldable design. In still other embodiments, display screen 805 can be a flexible display, located on a curved or foldable surface of terminal 800. Display screen 805 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 805 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0239] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0240] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 801 for processing, or input into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 800. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0241] Positioning component 808 is used to locate the current geographic location of terminal 800 to implement navigation or LBS (Location Based Service). Positioning component 808 can be based on the US GPS (Global Positioning System), China's Beidou system, Russia's Greninja system, or the European Union's Galileo system.

[0242] Power supply 809 is used to power various components in terminal 800. Power supply 809 can be AC ​​power, DC power, disposable batteries, or rechargeable batteries. When power supply 809 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0243] In some embodiments, the terminal 800 further includes one or more sensors 810 , including but not limited to: an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , an optical sensor 815 , and a proximity sensor 816 .

[0244] The accelerometer 811 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 800. For example, the accelerometer 811 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 801 can control the touch screen display 805 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 811. The accelerometer 811 can also be used to collect game or user motion data.

[0245] The gyroscope sensor 812 can detect the orientation and rotation angle of the terminal 800. It can work in conjunction with the accelerometer 811 to collect the user's 3D movements of the terminal 800. Based on the data collected by the gyroscope sensor 812, the processor 801 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0246] The pressure sensor 813 can be located on the side frame of the terminal 800 and / or below the touch screen display 805. When the pressure sensor 813 is located on the side frame of the terminal 800, it can detect the user's grip signal on the terminal 800. The processor 801 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 813. When the pressure sensor 813 is located below the touch screen display 805, the processor 801 controls the operable controls on the UI interface based on the user's pressure operation on the touch screen display 805. Operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0247] The optical sensor 815 is used to detect ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the touchscreen display 805 based on the ambient light intensity detected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the touchscreen display 805 is increased; when the ambient light intensity is low, the display brightness of the touchscreen display 805 is decreased. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 based on the ambient light intensity detected by the optical sensor 815.

[0248] Proximity sensor 816, also known as a distance sensor, is typically located on the front panel of terminal 800. Proximity sensor 816 is used to detect the distance between the user and the front of terminal 800. In one embodiment, when proximity sensor 816 detects that the distance between the user and the front of terminal 800 is gradually decreasing, processor 801 controls touchscreen display 805 to switch from the screen-on state to the screen-off state. When proximity sensor 816 detects that the distance between the user and the front of terminal 800 is gradually increasing, processor 801 controls touchscreen display 805 to switch from the screen-off state to the screen-on state.

[0249] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0250] If the electronic device is provided as a server, see Figure 9 , Figure 99 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 900 may vary significantly due to different configurations or performance, and may include a processor (central processing unit, CPU) 901 and a memory 902. The memory 902 stores at least one program code, which is loaded and executed by the processor 901 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server 900 may also have components such as a wired or wireless network interface, a keyboard, and input / output interfaces for input and output. The server 900 may also include other components for implementing device functions, which are not described in detail here.

[0251] If the electronic device is provided as a terminal and a server, the structural block diagrams of the terminal and the server can be respectively referred to Figure 8 and Figure 9 .

[0252] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the visual perception model training method in the above embodiment.

[0253] In an exemplary embodiment, a computer program product is also provided. The computer program product stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the visual perception model training method in the above embodiment.

[0254] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network may constitute a blockchain system.

[0255] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0256] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A visual perception model training method, characterized in that: The method comprises: Acquire a plurality of single-instance sample images, each of which includes an object; For each single-instance sample image, performing a first data enhancement process and a second data enhancement process on the single-instance sample image to obtain a first processed image and a second processed image; Selecting M single-instance sample images from the multiple single-instance sample images, and performing splicing processing on the M single-instance sample images to obtain a multi-instance sample image; wherein M is a positive integer greater than 1; performing feature extraction on the first processed image, the multi-instance sample image, the second processed image, and the M single-instance sample images, respectively, to obtain a first image feature, a second image feature, a third image feature, and M fourth image features; Performing model training based on the first image feature, the second image feature, the third image feature, and the M fourth image features to obtain a visual perception model; The first image feature includes: a first global feature and a first local feature; the third image feature includes: a second global feature and a second local feature; The performing model training based on the first image feature, the second image feature, the third image feature, and the M fourth image features to obtain a visual perception model includes: Constructing a first loss function based on the first global feature and the second global feature; Constructing a second loss function based on the first local feature and the second local feature; Constructing a third loss function based on the second image feature and the M fourth image features; Based on the first loss function, the second loss function and the third loss function, model training is performed to obtain the visual perception model.

2. The method according to claim 1, characterized in that The first data enhancement process includes: a first global enhancement process and a first local cropping process; the second data enhancement process includes: a second global enhancement process and a second local cropping process; the first processed image includes: a first global image and a first local image, and the second processed image includes: a second global image and a second local image; The performing a first data enhancement process and a second data enhancement process on the single instance sample image to obtain a first processed image and a second processed image respectively includes: Performing a first global enhancement process and a first local cropping process on the single instance sample image to obtain the first global image and the first local image; The single instance sample image is subjected to a second global enhancement process and a second local cropping process respectively to obtain the second global image and the second local image; wherein the first local image and the second local image have an overlapping area.

3. The method according to claim 1, characterized in that The step of performing splicing processing on the M single-instance sample images to obtain multiple-instance sample images includes: Performing a third global enhancement process on the M single-instance sample images respectively to obtain M third processed images; The M third processed images are spliced ​​together to obtain the multiple instance sample images.

4. The method according to claim 1, wherein The process of extracting features from the multiple instance sample images to obtain second image features includes: Performing feature extraction on the multiple instance sample images through a first residual network to obtain fifth image features; Performing average pooling processing on the fifth image feature to obtain M third sub-features; The second correction processing and the full connection processing are sequentially performed on the M third sub-features to obtain the second image feature.

5. The method according to claim 1, wherein The constructing a third loss function based on the second image feature and the M fourth image features includes: Constructing a first positive sample pair, the first positive sample pair including the second image feature and the M fourth image features; constructing a first negative sample pair, the first negative sample pair including the second image feature and the second global feature; Determining a first similarity and a second similarity; wherein the first similarity is used to measure the similarity between the second image feature and the M fourth image features, and the second similarity is used to measure the similarity between the second image feature and the second global feature; The third loss function is constructed based on the first similarity and the second similarity.

6. The method according to claim 5, characterized in that The process of determining the first similarity includes: For each fourth image feature, determining a third similarity between the second image feature and the fourth image feature, to obtain M third similarities; The sum of the M third similarities is determined as the first similarity.

7. The method according to claim 2, characterized in that The first local feature includes: N first sub-features, and the second local feature includes: N fourth sub-features; The constructing a second loss function based on the first local feature and the second local feature includes: Constructing a second positive sample pair, the second positive sample pair including the N first sub-features and the N fourth sub-features; constructing a second negative sample pair, the second negative sample pair including the N first sub-features and a third local feature, the third local feature being obtained by performing feature extraction on a fourth local image, the fourth local image being an image having no overlapping area with the first local image; Determining a fourth similarity and a fifth similarity; wherein the fourth similarity is used to measure the similarity between the N first sub-features and the N fourth sub-features, and the fifth similarity is used to measure the similarity between the N first sub-features and the third local feature; The second loss function is constructed based on the fourth similarity and the fifth similarity.

8. A visual perception model training device, characterized in that: The device comprises: An acquisition module, configured to acquire a plurality of single-instance sample images, each of which includes an object; A first processing module is configured to perform a first data enhancement process and a second data enhancement process on each single instance sample image, respectively, to obtain a first processed image and a second processed image; a second processing module, configured to select M single-instance sample images from the plurality of single-instance sample images, and perform splicing processing on the M single-instance sample images to obtain a multi-instance sample image; wherein M is a positive integer greater than 1; a feature extraction module, configured to perform feature extraction on the first processed image, the multi-instance sample image, the second processed image, and the M single-instance sample images, respectively, to obtain a first image feature, a second image feature, a third image feature, and M fourth image features; a training module, configured to perform model training based on the first image feature, the second image feature, the third image feature, and the M fourth image features to obtain a visual perception model, wherein the first image feature includes: a first global feature and a first local feature, and the third image feature includes: a second global feature and a second local feature; The performing model training based on the first image feature, the second image feature, the third image feature, and the M fourth image features to obtain a visual perception model includes: Constructing a first loss function based on the first global feature and the second global feature; Constructing a second loss function based on the first local feature and the second local feature; Constructing a third loss function based on the second image feature and the M fourth image features; Based on the first loss function, the second loss function and the third loss function, model training is performed to obtain the visual perception model.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the visual perception model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the visual perception model training method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the visual perception model training method according to any one of claims 1 to 7.

Images