Training method, device and medium for neural network generating intermediate features

Abstract

Disclosed are a training method, device, and medium for a neural network for generating intermediate features. The method includes: obtaining a native image and a first visual image; annotating the first visual image with information to obtain annotation information associated with a machine vision task; generating a first intermediate feature based on the native image via a to-be-trained intermediate feature generation network; generating a second visual image based on the first intermediate feature; generating a task execution result of the machine vision task based on the first intermediate feature; training the to-be-trained intermediate feature generation network based on the first visual image, the annotated information, the second visual image, and the task execution result; and determining the trained to-be-trained intermediate feature generation network as a trained intermediate feature generation network in response to the trained intermediate feature generation network satisfying a preset training end condition. The embodiments of the present disclosure can improve the execution effect of machine vision tasks.

Description

Technical Field

[0001] The present disclosure relates to driving technology, and more particularly to a training method, apparatus, and medium for a neural network for generating intermediate features. Background Art

[0002] Visual imaging and machine vision are key technologies in the field of autonomous driving. For example, in the case of object detection, image signal processing (ISP) technology is typically used to process the native image captured by the image sensor to generate a visual image, such as an RGB image. Target detection is then performed on this visual image. In RGB, R represents red, G represents green, and B represents blue. Summary of the Invention

[0003] The present disclosure addresses the technical issue of the inability of the ISP processing process from native images to visualized images to guarantee the performance of machine vision tasks. Embodiments of the present disclosure provide a training method, apparatus, and medium for a neural network for generating intermediate features.

[0004] According to one aspect of an embodiment of the present disclosure, a method for training a neural network for generating intermediate features is provided, comprising:

[0005] Acquire a native image and a first visualized image, wherein one of the native image and the first visualized image is converted from the other;

[0006] Annotating the first visual image to obtain annotation information associated with the machine vision task;

[0007] Based on the native image, generating a first intermediate feature through a to-be-trained intermediate feature generation network;

[0008] generating a second visual image based on the first intermediate feature;

[0009] generating a task execution result of the machine vision task based on the first intermediate feature;

[0010] Training the intermediate feature generation network to be trained based on the first visual image, the annotation information, the second visual image, and the task execution result;

[0011] In response to the trained intermediate feature generation network to be trained meeting a preset training end condition, the trained intermediate feature generation network to be trained is determined as a trained intermediate feature generation network.

[0012] According to another aspect of an embodiment of the present disclosure, a training apparatus for a neural network for generating intermediate features is provided, comprising:

[0013] an acquisition module, configured to acquire a native image and a first visualized image, wherein one of the native image and the first visualized image is converted from the other;

[0014] an information annotation module, configured to annotate the first visual image acquired by the acquisition module to obtain annotation information associated with the machine vision task;

[0015] A first generating module is configured to generate a first intermediate feature based on the native image acquired by the acquiring module via an intermediate feature generating network to be trained;

[0016] a second generating module, configured to generate a second visual image based on the first intermediate features generated by the first generating module;

[0017] a third generating module, configured to generate a task execution result of the machine vision task based on the first intermediate feature generated by the first generating module;

[0018] a training module, configured to train the intermediate feature generation network to be trained based on the first visual image acquired by the acquisition module, the annotation information annotated by the information annotation module, the second visual image generated by the second generation module, and the task execution result generated by the third generation module;

[0019] The determining module is configured to determine the intermediate feature generation network to be trained as a trained intermediate feature generation network in response to the intermediate feature generation network to be trained after being trained by the training module meeting a preset training end condition.

[0020] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program for executing the above-mentioned training method of the neural network for generating intermediate features.

[0021] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device including:

[0022] processor;

[0023] a memory for storing instructions executable by the processor;

[0024] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-mentioned training method of the neural network for generating intermediate features.

[0025] According to another aspect of the embodiments of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor, the above-mentioned training method of the neural network for generating intermediate features is performed.

[0026] Based on the training method, device, medium, electronic device and computer program product for a neural network for generating intermediate features provided by the above-mentioned embodiments of the present disclosure, in the neural network training stage, a native image and a first visual image belonging to the training data can be obtained, information annotation is performed on the first visual image, and annotation information associated with the machine vision task is obtained. Based on the native image, a first intermediate feature is generated through the intermediate feature generation network to be trained. The first intermediate feature can be used for visual imaging and machine vision tasks, thereby obtaining a second visual image and the task execution result of the machine vision task. The first visual image, the annotation information, the second visual image and the task execution result can be used together to train the intermediate feature generation network to be trained. In this way, in the neural network inference stage, it is only necessary to provide the native image captured by the image sensor to the trained intermediate feature generation network, and the trained intermediate feature generation network can perform calculations based on this to efficiently and reliably generate corresponding intermediate features. The generated intermediate features can be used to obtain the task execution result of the machine vision task. Therefore, in the embodiments of the present disclosure, in the neural network inference stage, the execution of the machine vision task does not need to rely on the visualized image, but can directly rely on the intermediate features obtained from the native image. This can avoid the information discard problem in the ISP processing process from the native image to the visualized image, and can also avoid the problem that the ISP processing process is mainly for ensuring the visual quality of the human eye without considering the performance of machine vision, which is conducive to improving the execution effect of the machine vision task.

[0027] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of a training method for a neural network for generating intermediate features provided by an exemplary embodiment of the present disclosure.

[0029] Figure 2 It is a flowchart of a training method for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0030] Figure 3-1 is a schematic diagram of a native image in an exemplary embodiment of the present disclosure.

[0031] Figure 3-2 4 is a schematic diagram of a multi-channel image obtained by rearranging a native image in an exemplary embodiment of the present disclosure.

[0032] Figure 4-1 Schematic diagram of a neural network capable of taking into account both visual imaging and machine vision tasks in an embodiment of the present disclosure.

[0033] Figure 4-2 is a schematic diagram of another neural network in an embodiment of the present disclosure that can take into account both visual imaging and machine vision tasks.

[0034] Figure 5 4 is a flowchart of a training method for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0035] Figure 6 It is a schematic diagram of obtaining a second visual image from target segmentation features through pixel recombination in an embodiment of the present disclosure.

[0036] Figure 7 It is a flowchart of a training method for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0037] Figure 8 It is a flowchart of a training method for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0038] Figure 9 It is a flowchart of a training method for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0039] Figure 10 It is a flowchart of a training method for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0040] Figure 11 FIG. 1 is a schematic diagram of operations performed in the training phase and the inference phase of a neural network in an exemplary embodiment of the present disclosure.

[0041] Figure 12 It is a structural diagram of a training device for a neural network for generating intermediate features provided by an exemplary embodiment of the present disclosure.

[0042] Figure 13 It is a structural diagram of a training device for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0043] Figure 14 It is a structural diagram of a training device for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0044] Figure 15 It is a structural diagram of a training device for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0045] Figure 16 It is a structural diagram of a training device for a neural network for generating intermediate features provided by another exemplary embodiment of the present disclosure.

[0046] Figure 17 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0048] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0049] Application Overview

[0050] Visual imaging and machine vision are key technologies in the field of autonomous driving. Visual images generated through visual imaging include, but are not limited to, RGB and YUV images; the Y in YUV represents brightness, while the U and V represent color. Machine vision tasks include, but are not limited to, object detection, object tracking, semantic segmentation, and trajectory prediction.

[0051] Taking the case of object detection as an example, machine vision tasks typically require processing the native image captured by the image sensor using ISP technology to generate a visual image. This visual image is then used for object detection. The native image captured by the image sensor is also referred to as the image sensor's raw data, including but not limited to RGB images and RGB-IR images. The IR in RGB-IR stands for infrared.

[0052] Similarly, when the machine vision task is other tasks, it is also necessary to first obtain a visualization image and then perform processing related to other tasks on the obtained visualization image.

[0053] In the process of implementing the present disclosure, the inventors found that there is information discarding in the ISP processing process from native images to visualized images, which can easily affect the execution effect of machine vision tasks. In addition, the ISP processing process is mainly to ensure the visual quality of the human eye and does not consider the performance of machine vision. Therefore, it is difficult to ensure the execution effect of machine vision tasks.

[0054] Exemplary Methods

[0055] Figure 1 It is a flowchart of a training method for a neural network for generating intermediate features provided by an exemplary embodiment of the present disclosure. Figure 1 The method shown may include step 110, step 120, step 130, step 140, step 150, step 160 and step 170, and each step is described below.

[0056] Step 110 : Acquire a native image and a first visualized image, wherein one of the native image and the first visualized image is converted from the other.

[0057] Optionally, real Raw data can be collected through an image sensor and converted into a visual image through ISP simulation (or other simulators or simulation algorithms). In this case, the real Raw data can be used as the native image in step 110, and the visual image converted from the real Raw data can be used as the first visual image in step 110.

[0058] Optionally, a real visual image can also be obtained (for example, a real visual image can be obtained from an image library), and the real visual image can be converted into Raw data through a processing method that is reverse to the above-mentioned ISP simulation. In this case, the real visual image can be used as the first visual image in step 110, and the Raw data converted from the real visual image can be used as the native image in step 110.

[0059] Step 120 : Annotate the first visual image with information to obtain annotation information associated with the machine vision task.

[0060] Optionally, an information annotation operation may be manually performed on the first visual image, where the information annotation operation includes but is not limited to a touch input operation, a mouse input operation, a keyboard input operation, and the like.

[0061] In step 120, in response to the information annotation operation, annotation information associated with the machine vision task may be added to the first visual image. If the machine vision task is a target detection task, the annotation information associated with the machine vision task may include: a respective annotation box and an annotation category for each target object (e.g., a pedestrian, a vehicle, etc.) in the first visual image; if the machine vision task is a semantic segmentation task, the annotation information associated with the machine vision task may include: a respective annotation category for each pixel in the first visual image.

[0062] Step 130: Generate a first intermediate feature based on the native image through the intermediate feature generation network to be trained.

[0063] It should be noted that the intermediate feature generation network to be trained may be an intermediate feature generation network to be trained, and the intermediate features generated by the intermediate feature generation network to be trained may be intermediate features that can take into account both visualization imaging and machine vision tasks.

[0064] In step 130, the original image can be provided as input to the intermediate feature generation network to be trained. The intermediate feature generation network to be trained can then perform operations based on the original image to generate first intermediate features. Optionally, the first intermediate features can be multi-dimensional features, such as 14-dimensional features, 18-dimensional features, 20-dimensional features, or 24-dimensional features. Of course, the dimensionality of the first intermediate features is not limited to this and is not listed here.

[0065] Step 140: Generate a second visual image based on the first intermediate feature.

[0066] In step 140 , a second visualization image may be generated by visualization imaging with reference to the first intermediate feature.

[0067] Step 150: Generate a task execution result of the machine vision task based on the first intermediate feature.

[0068] In step 150, the first intermediate feature may be referenced to obtain a task execution result of the machine vision task by executing the machine vision task. If the machine vision task is an object detection task, the task execution result may include: a detection box and a detection category for each target object in the first visual image; if the machine vision task is a semantic segmentation task, the task execution result may include: multiple predicted probability values ​​for each pixel in the first visual image corresponding to multiple predicted categories.

[0069] Step 160: training the intermediate feature generation network to be trained based on the first visual image, the annotation information, the second visual image, and the task execution result.

[0070] It should be noted that the first visualization image and the annotation information can be considered as the true value data during the neural network training, and the second visualization image and the task execution result can be considered as the predicted data during the neural network training. In step 160, the performance of the intermediate feature generation network to be trained can be evaluated in combination with the true value data and the predicted data. Based on the evaluation results, the parameters of the intermediate feature generation network to be trained can be updated to optimize the performance of the intermediate feature generation network to be trained, thereby realizing the training of the intermediate feature generation network to be trained.

[0071] Step 170 : In response to the trained intermediate feature generation network to be trained meeting a preset training end condition, the trained intermediate feature generation network to be trained is determined as a trained intermediate feature generation network.

[0072] It should be noted that when training the intermediate feature generation network to be trained, a large amount of sample data can be used, and each sample data includes a native image and a corresponding first visualization image. In this way, the above steps 110 to 160 can be executed for each sample data, and the process of executing the above steps 110 to 160 for each sample data can be considered as an iterative processing.

[0073] After several iterative processes, if it is detected at a certain moment that the trained intermediate feature generation network to be trained converges, it can be determined that the trained intermediate feature generation network to be trained meets the preset training end conditions. At this time, the trained intermediate feature generation network to be trained can be directly determined as the trained intermediate feature generation network.

[0074] Of course, the preset training end condition is not limited to this. For example, when the number of iterative processing reaches a preset number, it can also be determined that the trained intermediate feature generation network meets the preset training end condition.

[0075] Based on the training method for a neural network for generating intermediate features provided in the above-mentioned embodiment of the present disclosure, in the neural network training stage, a native image and a first visual image belonging to the training data can be obtained, information annotation is performed on the first visual image, and annotation information associated with the machine vision task is obtained. Based on the native image, a first intermediate feature is generated through the intermediate feature generation network to be trained. The first intermediate feature can be used for visual imaging and machine vision tasks, thereby obtaining a second visual image and the task execution result of the machine vision task. The first visual image, the annotation information, the second visual image and the task execution result can be used together for the training of the intermediate feature generation network to be trained. In this way, in the neural network inference stage, it is only necessary to provide the native image captured by the image sensor to the trained intermediate feature generation network, and the trained intermediate feature generation network can perform calculations based on this to efficiently and reliably generate corresponding intermediate features. The generated intermediate features can be used to obtain the task execution result of the machine vision task. Therefore, in the embodiments of the present disclosure, in the neural network inference stage, the execution of the machine vision task does not need to rely on the visualized image, but can directly rely on the intermediate features obtained from the native image. This can avoid the information discard problem in the ISP processing process from the native image to the visualized image, and can also avoid the problem that the ISP processing process is mainly for ensuring the visual quality of the human eye without considering the performance of machine vision, which is conducive to improving the execution effect of the machine vision task.

[0076] In an alternative example, Figure 2 As shown, step 130 includes step 1301 and step 1303.

[0077] Step 1301 : Rearrange the native image based on multiple color components included in the native image to obtain a multi-channel image.

[0078] Alternatively, image rearrangement may also be referred to as image pack or image rearrange.

[0079] In an optional embodiment, the multi-channel image is a four-channel image, the first channel of the multi-channel image includes: each element located in odd rows and odd columns in the native image, the second channel of the multi-channel image includes: each element located in odd rows and even columns in the native image, the third channel of the multi-channel image includes: each element located in even rows and odd columns in the native image, and the fourth channel of the multi-channel image includes: each element located in even rows and even columns in the native image.

[0080] like Figure 3-1 As shown, the original image can be an RGGB image (which includes three color components). In this case, Figure 3-1 Starting from the element in the first row and first column, the elements are extracted at intervals of 1 pixel along the width and height directions, and then all the extracted elements are arranged in the first channel of the multi-channel image. When arranging, it can be ensured that the relative positions of any two elements after arrangement are the same as those of the two elements in the first channel. Figure 3-1 The relative positions in are consistent.

[0081] In addition, you can Figure 3-1 Starting from the element in the first row and second column, elements are extracted at intervals of 1 pixel along the width and height directions, and then all the extracted elements are arranged in the second channel of the multi-channel image. When arranging, it can be ensured that the relative positions of any two elements after arrangement are the same as those of the two elements in the second channel. Figure 3-1 The relative positions in are consistent.

[0082] In a similar way, the corresponding elements can be arranged in the third and fourth channels of the multi-channel image, thereby obtaining Figure 3-2 The complete multi-channel image is shown.

[0083] In this embodiment, by rearranging the native image, a multi-channel image can be obtained efficiently and reliably. Compared with the native image, the height and width of the multi-channel image can be reduced by half. In this way, compared with directly processing the native image, processing the multi-channel image can better save computing power.

[0084] It should be noted that, based on the above implementation, the four channels in the multi-channel image can also exchange positions. For example, the first channel of the multi-channel image includes: each element located in the even rows and even columns of the native image, the second channel of the multi-channel image includes: each element located in the even rows and odd columns of the native image, the third channel of the multi-channel image includes: each element located in the odd rows and even columns of the native image, and the fourth channel of the multi-channel image includes: each element located in the odd rows and odd columns of the native image.

[0085] In some embodiments, a multi-channel image can be obtained by other methods besides image rearrangement. For example, if the original image is still an RGGB image, the original image can be interpolated to obtain a four-channel image, and then the obtained four-channel image can be downsampled, and the downsampling result of the obtained four-channel image can be used as the multi-channel image in step 1301.

[0086] Step 1303: Perform multi-scale feature extraction on the multi-channel image through the intermediate feature generation network to be trained to obtain the first intermediate feature.

[0087] In an optional implementation, the intermediate feature generation network to be trained may include: a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network. Thus, step 1303 may include:

[0088] Performing feature extraction on the multi-channel image using a first sub-network in the intermediate feature generation network to be trained to obtain a first feature map having the same height and width as the multi-channel image;

[0089] Performing feature extraction on the first feature map by a second subnetwork in the intermediate feature generation network to be trained to obtain a second feature map whose height and width are respectively smaller than the height and width of the first feature map;

[0090] Performing feature extraction on the second feature map through a third subnetwork in the intermediate feature generation network to be trained to obtain a third feature map with the same height, width, and number of channels as the first feature map;

[0091] The fourth subnetwork in the intermediate feature generation network to be trained performs feature extraction based on the first feature map and the third feature map to obtain a first intermediate feature whose height and width are respectively the same as the height and width of the first feature map.

[0092] like Figure 4-1 、 Figure 4-2As shown, the first subnetwork may include: a first convolutional layer (Convolution, Conv), a first rectified linear unit (Rectified linear unit, ReLU) layer, and a second convolutional layer; wherein the first convolutional layer may be a 2D convolutional layer with a convolution kernel size of 3 and a stride of 1; the second convolutional layer may be a 2D convolutional layer with a convolution kernel size of 1 and a stride of 1. The second subnetwork may include: a third convolutional layer, a second rectified linear unit layer, and a fourth convolutional layer; wherein the third convolutional layer may be a 2D convolutional layer with a convolution kernel size of 3 and a stride of 2; the fourth convolutional layer may be a 2D convolutional layer with a convolution kernel size of 1 and a stride of 1. The third subnetwork may include: a first upsampling layer, which may specifically be a bilinear interpolation layer. The fourth sub-network may include: an element-by-element operation layer, a fifth convolutional layer, and a third rectified linear unit layer; wherein the element-by-element operation performed by the element-by-element operation layer includes but is not limited to an element-by-element addition operation, an element-by-element multiplication operation, etc.; the fifth convolutional layer may be a 2D convolutional layer with a convolution kernel size of 1 and a stride of 1.

[0093] After the multi-channel image obtained by image rearrangement is provided to the first sub-network, the multi-channel image will sequentially undergo convolution at the first convolutional layer, activation at the first rectified linear unit, and convolution at the second convolutional layer, so that the first feature map can be obtained by feature extraction without changing the height and width. After the first feature map is provided to the second sub-network, the first feature map will sequentially undergo convolution at the third convolutional layer, activation at the second rectified linear unit, and convolution at the fourth convolutional layer, so that the second feature map can be obtained by feature extraction with reduced height and width (specifically, the height and width are reduced by half). After the second feature map is provided to the third sub-network, the second feature map will undergo upsampling at the first upsampling layer, so that the third feature map can be obtained by feature extraction with enlarged height and width (specifically, the width and height are doubled). The height, width, and number of channels of the third feature map can be the same as those of the first feature map, respectively. Since the height, width, and number of channels of the third feature map can be the same as the height, width, and number of channels of the first feature map, respectively, after the third feature map and the first feature map are provided to the fourth sub-network, the third feature map and the first feature map can undergo element-by-element correspondence operation at the element-by-element correspondence operation layer, and the element-by-element correspondence operation results of the third feature map and the first feature map will sequentially undergo convolution at the fifth convolutional layer and activation at the third rectified linear unit layer, so that the first intermediate feature can be obtained by feature extraction without changing the height and width.

[0094] In this embodiment, by generating the intermediate feature to be trained including the first sub-network to the fourth sub-network, multi-scale feature extraction of multi-channel images can be achieved efficiently and reliably. The overall structure of the intermediate feature generation network to be trained is simple and the training speed is fast.

[0095] Of course, the implementation of step 1303 is not limited to this. For example, the first to fourth subnetworks in the intermediate feature generation network to be trained can adopt more complex structures. For another example, the intermediate feature generation network to be trained can be a feature pyramid network (FPN). By providing a multi-channel image as input to the FPN, the FPN can generate feature maps of different scales and fuse them. The fusion result of the feature maps of different scales can be used as the first intermediate feature.

[0096] In the embodiments of the present disclosure, referring to the multiple color components included in the original image, the original image can be efficiently and reliably converted into a multi-channel image through image rearrangement, and then by using the intermediate feature generation network to be trained, multi-scale feature extraction of the multi-channel image can be efficiently and reliably achieved. The first intermediate feature obtained by multi-scale feature extraction can contain very rich semantic information. Using the first intermediate feature for neural network training is conducive to improving the performance of the intermediate feature generation network finally obtained by training.

[0097] In an optional example, the ratio of the height of the native image to the height of the first intermediate feature is a target ratio, the ratio of the width of the native image to the width of the first intermediate feature is a target ratio, and the ratio of the number of channels of the first intermediate feature to the preset number of channels is: N times the square of the target ratio, the target ratio is an integer greater than or equal to 2, and N is an integer greater than or equal to 2.

[0098] Optionally, the preset number of channels may be the number of channels of a conventional type of visualization image (eg, an RGB image). Thus, the preset number of channels may be 3.

[0099] Optionally, N may be 2, 3, 4 or an integer greater than 4, which are not listed here one by one.

[0100] Assuming that the width of the native image is represented by W, the height of the native image is represented by H, the target ratio is represented by K, and the preset number of channels is represented by C, the width of the first intermediate feature can be represented by W / K, the height can be represented by H / K, and the number of channels can be represented by N*K 2 .

[0101] like Figure 5 As shown, step 140 includes step 1401, step 1403 and step 1405.

[0102] Step 1401: Segment the first intermediate feature along the channel direction to obtain N segmented features.

[0103] In step 1401, the first intermediate feature can be switched into N equal parts along the channel direction to obtain N segmented features; wherein each segmented feature can be in the form of a feature map, the width of each segmented feature can be expressed as W / K, the height can be expressed as H / K, and the number of channels can be expressed as K 2 .

[0104] Step 1403: Determine the target segmentation feature from the N segmentation features according to the preset screening rules.

[0105] Optionally, the N segmentation features may be arranged sequentially. In step 1403, the first segmentation feature may be selected from the sequentially arranged N segmentation features as the target segmentation feature; or, the last segmentation feature may be selected from the sequentially arranged N segmentation features as the target segmentation feature; or, when N is greater than or equal to 3, a segmentation feature other than the first segmentation feature and the last segmentation feature may be selected from the sequentially arranged N segmentation features as the target segmentation feature.

[0106] Step 1405, pixel reorganization of target segmentation features ( Figure 4-1 ), generating a second visualization image with the same height and width as the original image and a preset number of channels.

[0107] It can be understood that Pixel shuffle is an upsampling method that can replace interpolation or deconvolution (Transposed Convolution). Pixel shuffle can obtain a high-resolution feature map from a low-resolution feature map by reorganizing multiple channels.

[0108] In an example, the target segmentation feature is Figure 6 As shown in the feature map on the left, the width of the target segmentation feature is 5, the height is 5, and the number of channels is 12. Through Pixel shuffle, we can get Figure 6 The second visualization image shown on the right side of the figure has a width of 10, a height of 10, and a channel number of 3. Figure 6It can be seen that before Pixel shuffle, a, b, c, d, e, f, g, h, i, j, k, and l are located at corresponding positions of different channels of the target segmentation feature, that is, corresponding to the 1*1*12 area in the target segmentation feature, and after Pixel shuffle, a, b, c, d, e, f, g, h, i, j, k, and l correspond to the 2*2*3 area in the upper left corner of the second visualization image. Through Pixel shuffle, 2x upsampling can be achieved, and a three-dimensional feature (that is, the second visualization image) can be output.

[0109] In the embodiments of the present disclosure, by combining the segmentation operation of the first intermediate feature along the channel direction, the screening operation of the target segmentation feature, and the pixel reorganization operation of the target segmentation feature, a visualization image with a size that meets the requirements can be obtained efficiently and reliably as the second visualization image.

[0110] In an optional example, the height of the first intermediate feature is smaller than the height of the native image, the width of the first intermediate feature is smaller than the width of the native image, and the number of channels of the first intermediate feature is greater than a preset number of channels.

[0111] It should be noted that, in the embodiment of the present disclosure, the first neural network can be used to generate a visual image. Optionally, the first neural network can be a convolutional neural network, and the first neural network can include: a fifth sub-network and a sixth sub-network. Figure 7 As shown, step 140 may include step 1407 and step 1409.

[0112] Step 1407: Perform feature extraction on the first intermediate feature through the fifth subnetwork in the first neural network for generating a visual image to obtain a fourth feature map having the same height and width as the first intermediate feature.

[0113] like Figure 4-2 As shown, the fifth sub-network may include: a sixth convolutional layer, a fourth rectified linear unit layer and a seventh convolutional layer; wherein the sixth convolutional layer may be a 2D convolutional layer with a convolution kernel size of 3 and a stride of 2; the seventh convolutional layer may be a 2D convolutional layer with a convolution kernel size of 1 and a stride of 1.

[0114] In step 1409 , the sixth subnetwork in the first neural network is used to extract features from the fourth feature map to obtain a second visualization image having the same height and width as the original image and a preset number of channels.

[0115] like Figure 4-2As shown, the sixth subnetwork may include: a second upsampling layer, an eighth convolutional layer and a fifth rectified linear unit layer; wherein the second upsampling layer may specifically be a bilinear interpolation layer; the eighth convolutional layer may be a 2D convolutional layer with a convolution kernel size of 1 and a stride of 1.

[0116] After providing the first intermediate feature to the fifth sub-network, the first intermediate feature will sequentially undergo convolution at the sixth convolutional layer, activation at the fourth rectified linear unit layer, and convolution at the seventh convolutional layer, so that the fourth feature map can be obtained by feature extraction without changing the height and width. After providing the fourth feature map to the sixth sub-network, the fourth feature map will undergo upsampling at the second upsampling layer, so that the fifth feature map can be obtained by feature extraction with amplified height and width (specifically, doubling both width and height). Thereafter, the fifth feature map can sequentially undergo convolution at the eighth convolutional layer and activation at the fifth rectified linear unit layer, so that the height and width can be obtained by feature extraction without changing the height and width but changing the number of channels, and the second visualization image with the same height and width as the original image and the preset number of channels can be obtained.

[0117] In the embodiment of the present disclosure, a first neural network including the fifth to sixth subnetworks can efficiently and reliably obtain a visualization image with a size that meets the requirements as the second visualization image. The overall structure of the first neural network is simple and easy to train.

[0118] Of course, the fifth sub-network and the sixth sub-network in the first neural network may also adopt a more complex structure, as long as the first neural network is used to obtain a second visualization image with a size that meets the requirements from the first intermediate feature.

[0119] In an alternative example, Figure 8 As shown, step 160 includes step 1601, step 1603, step 1605 and step 1607.

[0120] Step 1601: Determine a first loss value by comparing a first visualized image and a second visualized image.

[0121] In step 1601, by comparing the first visual image and the second visual image, the similarity between the first visual image and the second visual image can be calculated, and based on the similarity, a first loss value can be determined. Optionally, the difference between a preset value (e.g., 1) and the similarity can be calculated, and the calculated difference can be used as the first loss value. Of course, the method for determining the first loss value is not limited to this. It is sufficient to ensure that there is a negative correlation between the similarity and the first loss value. The embodiments of the present disclosure are not limited to this.

[0122] Step 1603: Determine the second loss value by comparing the annotation information and the task execution result.

[0123] Assuming that the machine vision task is a target detection task, the annotation information associated with the machine vision task includes: the respective annotation box and annotation category of each target object in the first visualization image, and the task execution result of the machine vision task includes: the respective detection box and detection category of each target object in the first visualization image. For each target object, the object loss value of the target object can be determined by combining the intersection-over-union ratio between the detection box and the annotation box corresponding to the target object, and whether the detection category and the annotation category corresponding to the target object are the same. By summing or averaging the object loss values ​​corresponding to each target object, a second loss value can be obtained.

[0124] Assuming that the machine vision task is a semantic segmentation task, the annotation information associated with the machine vision task includes: the respective annotation categories of each pixel in the first visualization image, and the task execution result of the machine vision task includes: each pixel in the first visualization image corresponds to multiple prediction probability values ​​of multiple prediction categories. Then, for each pixel, a prediction category that matches the annotation category of the pixel in multiple prediction categories can be determined (the prediction category can be the same category as the annotation category), and the prediction probability value corresponding to the determined prediction category can be selected from the multiple prediction probability values ​​corresponding to the pixel. With reference to the selected prediction probability value, the pixel loss value of the pixel can be determined (the pixel loss value and the selected prediction probability value can be negatively correlated), and the second loss value can be obtained by summing or averaging the pixel loss values ​​corresponding to each pixel.

[0125] Step 1605: Determine a model loss value based on the first loss value and the second loss value.

[0126] In step 1605, the first loss value and the second loss value can be directly summed to obtain the model loss value; or, the first loss value and the second loss value can be averaged to obtain the model loss value; or, the first loss value and the second loss value can be weighted averaged to obtain the model loss value.

[0127] Step 1607: Use the model loss value to train the intermediate feature generation network to be trained.

[0128] In step 1607, with reference to the model loss value, a gradient descent method (such as a stochastic gradient descent method, a steepest gradient descent method, etc.) can be used to update the parameters of the intermediate feature generation network to be trained, thereby achieving the training of the intermediate feature generation network to be trained.

[0129] In an embodiment of the present disclosure, a first loss value can be determined by comparing the first visual image and the second visual image. The first loss value can effectively characterize the reliability of the visual image generated when the intermediate feature generation network to be trained is used for visual image generation. A second loss value can be determined by comparing the annotation information and the task execution result. The second loss value can effectively characterize the reliability of the task execution result generated when the intermediate feature generation network to be trained is used for machine vision tasks. By using the model loss value obtained based on the first loss value and the second loss value for the training of the intermediate feature generation network to be trained, the reliability of the intermediate feature generation network finally obtained by training can be improved through back propagation, so that the intermediate feature generation network finally obtained by training can be effectively adapted to visual imaging and machine vision tasks.

[0130] In an alternative example, Figure 9 As shown, step 140 includes step 1413.

[0131] Step 1413: Generate a second visual image based on the first intermediate feature via a first neural network for generating a visual image.

[0132] It should be noted that the structure of the first neural network refers to Figure 7 In step 1413, the first intermediate feature is provided as input to the first neural network, and the first neural network can perform operations based on the first intermediate feature to generate a second visual image.

[0133] Step 160, including step 1609.

[0134] Step 1609: Based on the first visual image, the annotation information, the second visual image, and the task execution result, the intermediate feature generation network to be trained and the first neural network are trained.

[0135] In the embodiments of the present disclosure, the model loss value mentioned above can be obtained by combining the first visual image, annotation information, the second visual image and the task execution result. The model loss value can be used to adjust the parameters of the intermediate feature generation network to be trained and the first neural network, which is conducive to ensuring the reliability of the visual image generated by the trained first neural network.

[0136] In an alternative example, Figure 10 As shown, step 150 includes step 1501.

[0137] Step 1501: Generate a task execution result based on the first intermediate feature via a second neural network for performing a machine vision task.

[0138] It should be noted that if the machine vision task is a target detection task, the second neural network can be a conventional target detection network (also called a target detector); if the machine vision task is a semantic segmentation task, the second neural network can be a conventional semantic segmentation network.

[0139] In step 1501, the first intermediate feature only needs to be provided as input to the second neural network, and the second neural network can perform calculations based on it to generate a task execution result of the machine vision task.

[0140] Step 160 includes step 1611.

[0141] Step 1611: Based on the first visual image, the annotation information, the second visual image, and the task execution result, the intermediate feature generation network to be trained and the second neural network are trained.

[0142] In the embodiments of the present disclosure, the model loss value mentioned above can be obtained by combining the first visual image, annotation information, the second visual image and the task execution result. The model loss value can be used together to adjust the parameters of the intermediate feature generation network to be trained and the second neural network, which is conducive to ensuring the reliability of the task execution results generated by the trained second neural network for the machine vision task.

[0143] In an alternative example, Figure 11As shown, during the neural network training phase, real Raw data can be collected through an image sensor and converted into a visual image (equivalent to the first visual image mentioned above) through ISP simulation. The converted visual image can be considered as an RGB label. Information annotation can be performed on the converted visual image to obtain annotation information for the target detection task. The real Raw data can be provided to the intermediate feature generation network (which currently belongs to the intermediate feature generation network to be trained), and the first intermediate feature generated by the intermediate feature generation network can be provided to the target detection network. The target detection network can obtain the target detection result based on the first intermediate feature, and the target detection result can be used together with the annotation information for Loss calculation (corresponding to the calculation of the model loss value mentioned above). In addition, the first intermediate feature generated by the intermediate feature generation network can also be used to generate a visual image (equivalent to the second visual image mentioned above). For example, the first intermediate feature can be a 24-dimensional feature, then the first 12-dimensional feature can be extracted from the 24-dimensional feature (equivalent to the target segmentation feature mentioned above), and a visual image can be generated by performing Pixelshuffle on the extracted first 12-dimensional features, or by inputting the extracted first 12-dimensional features into the first neural network to generate a visual image. The generated visual image can be used together with the RGB label for Loss calculation (corresponding to the calculation of the model loss value mentioned above). Based on the model loss value obtained by Loss calculation, the parameters of the intermediate feature generation network, the first neural network and the target detection network can be updated through back propagation.

[0144] During the neural network inference stage, it is only necessary to provide the real Raw data to the intermediate feature generation network (which currently belongs to the trained intermediate feature generation network), and the intermediate feature generation network can generate the second intermediate feature. Based on the second intermediate feature, the trained target detection network can efficiently and reliably output the target detection results. In addition, based on the second intermediate feature, the trained first neural network can efficiently and reliably generate visual images.

[0145] In summary, the embodiments of the present disclosure can take into account both visualization imaging and machine vision tasks, and can ensure the execution effect of machine vision tasks.

[0146] Any of the neural network training methods for generating intermediate features provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the neural network training methods for generating intermediate features provided in the embodiments of the present disclosure can be executed by a processor, such as a processor that executes any of the neural network training methods for generating intermediate features mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be further described below.

[0147] Exemplary devices

[0148] Figure 12 It is a structural diagram of a training device for a neural network for generating intermediate features provided by an exemplary embodiment of the present disclosure. Figure 12 The device shown includes an acquisition module 1210 , an information annotation module 1220 , a first generation module 1230 , a second generation module 1240 , a third generation module 1250 , a training module 1260 and a determination module 1270 .

[0149] An acquisition module 1210 is configured to acquire a native image and a first visualized image, wherein one of the native image and the first visualized image is converted from the other;

[0150] An information annotation module 1220 is configured to annotate the first visual image acquired by the acquisition module 1210 to obtain annotation information associated with the machine vision task;

[0151] A first generating module 1230 is configured to generate a first intermediate feature based on the native image acquired by the acquiring module 1210 via an intermediate feature generation network to be trained;

[0152] A second generating module 1240 is configured to generate a second visual image based on the first intermediate features generated by the first generating module 1230;

[0153] A third generating module 1250 is configured to generate a task execution result of the machine vision task based on the first intermediate feature generated by the first generating module 1230;

[0154] A training module 1260 is configured to train the intermediate feature generation network to be trained based on the first visual image acquired by the acquisition module 1210, the annotation information annotated by the information annotation module 1220, the second visual image generated by the second generation module 1240, and the task execution result generated by the third generation module 1250;

[0155] The determination module 1270 is configured to determine the trained intermediate feature generation network to be trained as a trained intermediate feature generation network in response to the intermediate feature generation network to be trained after being trained by the training module 1260 meeting a preset training end condition.

[0156] In an alternative example, Figure 13 As shown, the first generation module 1230 includes:

[0157] An image rearrangement submodule 12301 is configured to rearrange the native image acquired by the acquisition module 1210 based on multiple color components included in the native image to obtain a multi-channel image.

[0158] The feature extraction submodule 12303 is used to perform multi-scale feature extraction on the multi-channel image obtained by the image rearrangement submodule 12301 through the intermediate feature generation network to be trained to obtain the first intermediate feature.

[0159] In an optional example, the feature extraction submodule 12303 includes:

[0160] A first feature extraction unit is configured to perform feature extraction on the multi-channel image obtained by the image rearrangement submodule 12301 using a first sub-network in the intermediate feature generation network to be trained, to obtain a first feature map having the same height and width as the multi-channel image;

[0161] A second feature extraction unit is configured to perform feature extraction on the first feature map obtained by the first feature extraction unit through a second subnetwork in the intermediate feature generation network to be trained, so as to obtain a second feature map whose height and width are respectively smaller than the height and width of the first feature map;

[0162] a third feature extraction unit, configured to perform feature extraction on the second feature map obtained by the second feature extraction unit through a third subnetwork in the intermediate feature generation network to be trained, to obtain a third feature map having the same height, width, and number of channels as the first feature map;

[0163] The fourth feature extraction unit is used to perform feature extraction based on the first feature map obtained by the first feature extraction unit and the third feature map obtained by the third feature extraction unit through the fourth subnetwork in the intermediate feature generation network to be trained, so as to obtain a first intermediate feature whose height and width are respectively the same as the height and width of the first feature map.

[0164] In an optional example, the multi-channel image is a four-channel image, the first channel of the multi-channel image includes: each element located in odd rows and odd columns of the native image, the second channel of the multi-channel image includes: each element located in odd rows and even columns of the native image, the third channel of the multi-channel image includes: each element located in even rows and odd columns of the native image, and the fourth channel of the multi-channel image includes: each element located in even rows and even columns of the native image.

[0165] In an optional example, a ratio of a height of the native image to a height of the first intermediate feature is a target ratio, a ratio of a width of the native image to a width of the first intermediate feature is a target ratio, and a ratio of the number of channels of the first intermediate feature to a preset number of channels is: N times the square of the target ratio, the target ratio is an integer greater than or equal to 2, and N is an integer greater than or equal to 2;

[0166] like Figure 14As shown, the second generation module 1240 includes:

[0167] A sub-slicing module 12401 is configured to slice the first intermediate feature generated by the first generating module 1230 along the channel direction to obtain N sliced ​​features;

[0168] The first determination submodule 12403 is configured to determine a target segmentation feature from the N segmentation features obtained by the segmentation submodule 12401 according to a preset screening rule;

[0169] The first generating submodule 12405 is used to perform pixel reorganization on the target segmentation features determined by the first determining submodule 12403 to generate a second visualization image having the same height and width as the original image and a preset number of channels.

[0170] In an optional example, a height of the first intermediate feature is smaller than a height of the native image, a width of the first intermediate feature is smaller than a width of the native image, and a number of channels of the first intermediate feature is greater than a preset number of channels;

[0171] like Figure 15 As shown, the second generation module 1240 includes:

[0172] A first feature extraction submodule 12407 is configured to perform feature extraction on the first intermediate feature generated by the first generation module 1230 using a fifth subnetwork in the first neural network for generating a visual image, to obtain a fourth feature map having the same height and width as the first intermediate feature;

[0173] The second feature extraction submodule 12409 is used to perform feature extraction on the fourth feature map obtained by the first feature extraction submodule 12407 through the sixth subnetwork in the first neural network, to obtain a second visualization image whose height and width are respectively the same as the height and width of the native image and the number of channels is a preset number of channels.

[0174] In an alternative example, Figure 16 As shown, the training module 1260 includes:

[0175] A second determining submodule 12601 is configured to determine a first loss value by comparing the first visual image acquired by the acquiring module 1210 with the second visual image generated by the second generating module 1240;

[0176] The third determining submodule 12603 is configured to determine a second loss value by comparing the annotation information annotated by the information annotation module 1220 and the task execution result generated by the third generating module 1250;

[0177] A fourth determining submodule 12605 is configured to determine a model loss value based on the first loss value determined by the second determining submodule 12601 and the second loss value determined by the second determining submodule 12601;

[0178] The training submodule 12607 is used to train the intermediate feature generation network to be trained using the model loss value determined by the fourth determination submodule 12605.

[0179] In an optional example, the second generating module 1240 includes:

[0180] a fifth determining submodule, configured to determine a first neural network for generating a visual image;

[0181] A second generating submodule, configured to generate a second visual image based on the first intermediate features generated by the first generating module 1230 and the first neural network determined by the fifth determining submodule;

[0182] The training module 1260 is configured to:

[0183] Based on the first visualization image acquired by the acquisition module 1210, the annotation information annotated by the information annotation module 1220, the second visualization image generated by the second generation module 1240 and the task execution result generated by the third generation module 1250, the intermediate feature generation network to be trained and the first neural network determined by the fifth determination submodule are trained.

[0184] In an optional example, the third generating module 1250 includes:

[0185] a sixth determination submodule, configured to determine a second neural network for performing a machine vision task;

[0186] A third generating submodule, configured to generate a task execution result based on the first intermediate feature generated by the first generating module 1230 and the second neural network determined by the sixth determining submodule;

[0187] The training module 1260 is configured to:

[0188] Based on the first visualization image acquired by the acquisition module 1210, the annotation information annotated by the information annotation module 1220, the second visualization image generated by the second generation module 1240 and the task execution result generated by the third generation module 1250, the intermediate feature generation network to be trained and the second neural network determined by the sixth determination submodule are trained.

[0189] In the device of the present disclosure, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0190] The beneficial technical effects corresponding to the exemplary embodiment of this device can be found in the corresponding beneficial technical effects of the above exemplary method part, which will not be repeated here.

[0191] Exemplary electronic devices

[0192] Figure 17 17 is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 1700 includes one or more processors 1710 and a memory 1720 .

[0193] The processor 1710 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1700 to perform desired functions.

[0194] The memory 1720 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1710 may execute the one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0195] In one example, the electronic device 1700 may further include an input device 1730 and an output device 1740 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0196] The input device 1730 may also include, for example, a keyboard, a mouse, and the like.

[0197] The output device 1740 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0198] Of course, to simplify, Figure 17 Only some of the components related to the present disclosure in the electronic device 1700 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1700 may further include any other appropriate components.

[0199] Exemplary computer program products and computer-readable storage media

[0200] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to execute the steps of the training method for a neural network for generating intermediate features according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0201] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0202] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the training method for a neural network for generating intermediate features according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0203] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0204] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present disclosure. The specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. The above details do not limit the present disclosure to necessarily being implemented using the above specific details.

[0205] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A method for training a neural network for generating intermediate features, comprising: Acquire a native image and a first visualized image, wherein one of the native image and the first visualized image is converted from the other; Annotating the first visual image to obtain annotation information associated with the machine vision task; Based on the native image, generating a first intermediate feature through a to-be-trained intermediate feature generation network; generating a second visual image based on the first intermediate feature; generating a task execution result of the machine vision task based on the first intermediate feature; Training the intermediate feature generation network to be trained based on the first visual image, the annotation information, the second visual image, and the task execution result; In response to the trained intermediate feature generation network to be trained meeting a preset training end condition, the trained intermediate feature generation network to be trained is determined as a trained intermediate feature generation network.

2. The method according to claim 1, wherein The step of generating a first intermediate feature based on the native image by using a to-be-trained intermediate feature generation network includes: Based on multiple color components included in the native image, performing image rearrangement on the native image to obtain a multi-channel image; Multi-scale feature extraction is performed on the multi-channel image through the intermediate feature generation network to be trained to obtain the first intermediate feature.

3. The method according to claim 2, wherein: The step of generating the intermediate feature to be trained by extracting multi-scale features from the multi-channel image to obtain the first intermediate feature includes: Performing feature extraction on the multi-channel image through a first sub-network in the intermediate feature generation network to be trained to obtain a first feature map with the same height and width as the multi-channel image; Performing feature extraction on the first feature map by a second subnetwork in the intermediate feature generation network to be trained to obtain a second feature map whose height and width are respectively smaller than the height and width of the first feature map; Performing feature extraction on the second feature map using a third subnetwork in the intermediate feature generation network to be trained to obtain a third feature map having the same height, width, and number of channels as the first feature map; The fourth subnetwork in the intermediate feature generation network to be trained is used to perform feature extraction based on the first feature map and the third feature map to obtain the first intermediate feature with the same height and width as the first feature map.

4. The method according to claim 2, wherein: The multi-channel image is a four-channel image, the first channel of the multi-channel image includes: each element located in odd rows and odd columns of the native image, the second channel of the multi-channel image includes: each element located in odd rows and even columns of the native image, the third channel of the multi-channel image includes: each element located in even rows and odd columns of the native image, and the fourth channel of the multi-channel image includes: each element located in even rows and even columns of the native image.

5. The method according to claim 1, wherein A ratio of the height of the native image to the height of the first intermediate feature is a target ratio, a ratio of the width of the native image to the width of the first intermediate feature is the target ratio, and a ratio of the number of channels of the first intermediate feature to the preset number of channels is N times the square of the target ratio, where the target ratio is an integer greater than or equal to 2, and N is an integer greater than or equal to 2; Generating a second visual image based on the first intermediate feature includes: Slicing the first intermediate feature along the channel direction to obtain N sliced ​​features; Determine the target segmentation feature from the N segmentation features according to the preset screening rules; Pixel reorganization is performed on the target segmentation features to generate a second visualization image whose height and width are respectively the same as the height and width of the original image and whose number of channels is the preset number of channels.

6. The method according to claim 1, wherein The height of the first intermediate feature is smaller than the height of the native image, the width of the first intermediate feature is smaller than the width of the native image, and the number of channels of the first intermediate feature is greater than a preset number of channels; Generating a second visual image based on the first intermediate feature includes: performing feature extraction on the first intermediate feature using a fifth subnetwork in the first neural network for generating a visual image, to obtain a fourth feature map having the same height and width as the first intermediate feature; The sixth subnetwork in the first neural network is used to extract features from the fourth feature map to obtain a second visualization image whose height and width are respectively the same as the height and width of the native image and whose number of channels is a preset number of channels.

7. The method according to any one of claims 1 to 6, wherein The training of the intermediate feature generation network to be trained based on the first visual image, the annotation information, the second visual image, and the task execution result includes: determining a first loss value by comparing the first visual image and the second visual image; Determining a second loss value by comparing the annotation information and the task execution result; Determining a model loss value based on the first loss value and the second loss value; The model loss value is used to train the intermediate feature generation network to be trained.

8. The method according to any one of claims 1 to 4, wherein Generating a second visual image based on the first intermediate feature includes: generating a second visual image based on the first intermediate feature via a first neural network for generating a visual image; The training of the intermediate feature generation network to be trained based on the first visual image, the annotation information, the second visual image, and the task execution result includes: Based on the first visual image, the annotation information, the second visual image and the task execution result, the intermediate feature generation network to be trained and the first neural network are trained.

9. The method according to any one of claims 1 to 6, wherein Generating a task execution result of the machine vision task based on the first intermediate feature includes: generating, based on the first intermediate feature, the task execution result via a second neural network for executing the machine vision task; The training of the intermediate feature generation network to be trained based on the first visual image, the annotation information, the second visual image, and the task execution result includes: Based on the first visual image, the annotation information, the second visual image and the task execution result, the intermediate feature generation network to be trained and the second neural network are trained.

10. A training apparatus for a neural network for generating intermediate features, comprising: an acquisition module, configured to acquire a native image and a first visualized image, wherein one of the native image and the first visualized image is converted from the other; an information annotation module, configured to annotate the first visual image acquired by the acquisition module to obtain annotation information associated with the machine vision task; A first generating module is configured to generate a first intermediate feature based on the native image acquired by the acquiring module via an intermediate feature generating network to be trained; a second generating module, configured to generate a second visual image based on the first intermediate features generated by the first generating module; a third generating module, configured to generate a task execution result of the machine vision task based on the first intermediate feature generated by the first generating module; a training module, configured to train the intermediate feature generation network to be trained based on the first visual image acquired by the acquisition module, the annotation information annotated by the information annotation module, the second visual image generated by the second generation module, and the task execution result generated by the third generation module; The determining module is configured to determine the intermediate feature generation network to be trained as a trained intermediate feature generation network in response to the intermediate feature generation network to be trained after being trained by the training module meeting a preset training end condition.

11. A computer-readable storage medium storing a computer program for executing the training method for a neural network for generating intermediate features according to any one of claims 1 to 9.

12. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the training method of the neural network for generating intermediate features described in any one of claims 1-9.

Images