Neural network generation, image processing method, device, electronic device and medium

Abstract

The present disclosure provides a method, device, electronic device and medium for generating a neural network, wherein the method for generating a neural network includes: performing a deformation correction transformation on a sample image to obtain a corrected image corresponding to the sample image; and training a neural network to be trained using the corrected image corresponding to the sample image to obtain a target neural network. The present disclosure performs a correction transformation on a sample image and then uses the obtained corrected image to train the neural network to be trained, thereby reducing the influence of the deformation of the target object in the image on the training result of the neural network and improving the accuracy of the target neural network.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a neural network generation, image processing method, device, electronic device and medium. Background Art

[0002] Object detection and semantic segmentation are two important image processing tasks in computer vision. Object detection can detect the target object in the image and obtain the specific location of the target object in the image; semantic segmentation can semantically classify the pixels in the image in the pixel dimension.

[0003] The image processing method obtains the target detection result or the semantic segmentation result with low accuracy. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a method, device, electronic device and medium for generating a neural network, processing an image.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a neural network, including: performing a deformation correction transformation on a sample image to obtain a corrected image corresponding to the sample image; and training a neural network to be trained using the corrected image corresponding to the sample image to obtain a target neural network.

[0006] In this way, by performing a correction transformation on the sample image and then using the obtained corrected image to train the neural network to be trained, the influence of the deformation of the target object in the image on the training result of the neural network can be reduced, thereby improving the accuracy of the target neural network.

[0007] In a possible implementation, a first pixel point in a corrected image obtained by performing a deformation correction transformation on the sample image is determined; for each first pixel point in the corrected image and the correction transformation relationship information between the sample image and the corresponding corrected image, a second pixel point in the sample image corresponding to the first pixel point is determined; wherein the correction transformation relationship includes: the relationship between the focal length, the position of the first pixel point in the corrected image, and the position of the second pixel point in the sample image; based on the pixel value of the second pixel point corresponding to the first pixel point, the pixel value of the first pixel point is determined; and based on the pixel values ​​of each first pixel point, the corrected image is generated.

[0008] In a possible implementation, the determining of the first pixel point in the corrected image obtained by performing a deformation correction transformation on the sample image includes: determining the position information of the corner pixel points in the corrected image based on the position information of the corner pixel points in the sample image and the correction transformation relationship information; and determining the first pixel point located in the corrected image based on the position information of the corner pixel points in the corrected image.

[0009] In a possible implementation, it also includes: generating the correction transformation relationship information in the following manner: constructing a rectangular coordinate system with the focus of the camera as the origin; and establishing a spherical coordinate system with the optical center of the camera as the origin and the focal length of the camera as the radius; determining the first projection point of the target object in the rectangular coordinate system and the second projection point in the spherical coordinate system based on the incident angle of the target object relative to the optical axis of the camera; based on the first coordinate value of the first projection point in the rectangular coordinate system and the second coordinate value of the second projection point in the spherical coordinate system, generating the conversion relationship information between the first projection point and the second projection point as the correction transformation relationship information.

[0010] In a possible implementation, the sample image includes a source domain sample image and a target domain sample image; performing a deformation correction transformation on the sample image to obtain a corrected image corresponding to the sample image includes: performing a deformation correction transformation on the source domain sample image and the target domain sample image respectively to obtain a source domain corrected image and a target domain corrected image.

[0011] In this way, deformation correction transformation is performed on the source domain sample image and the target domain sample image respectively, and the results of the deformation correction transformation are used to train the neural network, thereby reducing the influence of the deformation generated when the target object is projected onto the image on the feature alignment accuracy between the source domain and the target domain, thereby improving the accuracy of the neural network.

[0012] In a possible implementation, the method further includes: performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain corrected annotation information.

[0013] In a possible implementation, in the case where the annotation information includes semantic information of each pixel in a source domain sample image, performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain corrected annotation information includes: generating an annotation image based on the semantic information of each pixel in the source domain sample image; performing a deformation correction transformation on the annotation image to obtain a corrected annotation image of the annotation image; and obtaining the corrected annotation information based on the corrected annotation image.

[0014] In a possible implementation, in the case where the annotation information includes position information of the target object in the source domain sample image, the position information includes: coordinate values ​​of corner points of a detection box of the target object in the source domain sample image; performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain corrected annotation information includes: determining the coordinate values ​​of the corner points of the detection box in the source domain corrected image based on the coordinate values ​​of the corner points of the detection box in the source domain original image and correction transformation relationship information; and obtaining the corrected annotation information based on the coordinate values ​​of the corner points of the detection box in the source domain corrected image.

[0015] In a possible implementation, the corrected image corresponding to the sample image is used to train the neural network to obtain the target neural network, including: using the source domain corrected image, the target domain corrected image, and the corrected annotation information to train the neural network to obtain the target neural network; the using the source domain corrected image, the target domain corrected image, and the corrected annotation information to train the neural network to obtain the target neural network, including: using the neural network to be trained to process the source domain corrected image and the target domain corrected image to obtain a first result corresponding to the source domain corrected image and a second result corresponding to the target domain corrected image; determining a model loss based on the first result, the second result, the corrected annotation information, and the weight values ​​corresponding to each first pixel point in the source domain corrected image; and using the model loss to train the neural network to obtain the target neural network.

[0016] In a possible implementation, the method further includes: for each first pixel in the source domain corrected image, determining a weight value of the first pixel based on a coordinate value of a second pixel corresponding to the first pixel in the source domain sample image, and a coordinate value of a second pixel corresponding to an adjacent first pixel of the first pixel in the source domain sample image.

[0017] In a second aspect, an embodiment of the present disclosure further provides an image processing method, comprising: acquiring an image to be processed; performing a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed; processing the target corrected image using a target neural network generated by a neural network generation method as described in any one of the first aspects to obtain an intermediate processing result of the target corrected image; and obtaining a target processing result of the image to be processed based on the intermediate processing result.

[0018] In a possible implementation, the deformation correction transformation is performed on the image to be processed to obtain a target corrected image of the image to be processed, including: determining a first target pixel point in the target corrected image obtained by performing the deformation correction transformation on the image to be processed; determining a second target pixel point in the image to be processed corresponding to the target pixel point for each first target pixel point in the target corrected image and the correction transformation relationship information between the image to be processed and the corresponding target corrected image; determining a pixel value of the first target pixel point based on the pixel value of the second target pixel point corresponding to the first target pixel point; and generating the target corrected image based on the pixel values ​​of each first target pixel point.

[0019] In a possible implementation, the determining of the first target pixel point in the target corrected image obtained by performing a deformation correction transformation on the image to be processed includes: determining the position information of the corner pixel points in the target corrected image in the target corrected image based on the position information of the corner pixel points in the image to be processed and the correction transformation relationship information; and determining the first target pixel point located in the target corrected image based on the position information of the corner pixel points in the target corrected image in the target corrected image.

[0020] In a possible implementation, obtaining a target processing result of the image to be processed based on the intermediate processing result includes: performing an inverse deformation correction transformation on the intermediate processing result to obtain the target processing result of the image to be processed.

[0021] In a possible implementation, the target processing result includes: a semantic segmentation result of the image to be processed, and / or a target detection result of the image to be processed.

[0022] In a third aspect, the embodiments of the present disclosure also provide a device for generating a neural network, including: a first transformation module, used to perform a deformation correction transformation on a sample image to obtain a corrected image corresponding to the sample image; a training module, used to train a neural network to be trained using the corrected image corresponding to the sample image to obtain a target neural network.

[0023] In one possible implementation, the first transformation module, when performing a deformation correction transformation on a sample image, is used to: determine a first pixel point in a corrected image obtained by performing a deformation correction transformation on the sample image; determine a second pixel point in the sample image corresponding to the first pixel point for each first pixel point in the corrected image and correction transformation relationship information between the sample image and the corresponding corrected image; determine a pixel value of the first pixel point based on the pixel value of the second pixel point corresponding to the first pixel point; and generate the corrected image based on the pixel values ​​of each first pixel point.

[0024] In a possible implementation, the first transformation module, when determining the first pixel point in the corrected image obtained by performing a deformation correction transformation on the sample image, is used to: determine the position information of the corner pixel points in the corrected image based on the position information of the corner pixel points in the sample image and the correction transformation relationship information; and determine the first pixel point located in the corrected image based on the position information of the corner pixel points in the corrected image.

[0025] In a possible implementation, it also includes: a relationship determination module, which is used to generate the correction transformation relationship information in the following manner: construct a rectangular coordinate system with the focus of the camera as the origin; and, establish a spherical coordinate system with the optical center of the camera as the origin and the focal length of the camera as the radius; based on the incident angle of the target object relative to the optical axis of the camera, determine the first projection point of the target object in the rectangular coordinate system and the second projection point in the spherical coordinate system; based on the first coordinate value of the first projection point in the rectangular coordinate system and the second coordinate value of the second projection point in the spherical coordinate system, generate the transformation relationship information between the first projection point and the second projection point; use the transformation relationship information between the first projection point and the second projection point as the correction transformation relationship information.

[0026] In a possible implementation, the sample image includes a source domain sample image and a target domain sample image; and the first transformation module, when performing a deformation correction transformation on the sample image to obtain a corrected image corresponding to the sample image, is used to:

[0027] The source domain sample image and the target domain sample image are respectively subjected to deformation correction transformation to obtain the source domain corrected image and the target domain corrected image.

[0028] In a possible implementation manner, the first transformation module is further configured to: perform a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain corrected annotation information.

[0029] In a possible implementation, in the case where the annotation information includes: semantic information of each pixel in a source domain sample image, the first transformation module, when performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain corrected annotation information, is used to: generate an annotation image based on the semantic information of each pixel in the source domain sample image; perform a deformation correction transformation on the annotation image to obtain a corrected annotation image of the annotation image; and obtain the corrected annotation information based on the corrected annotation image.

[0030] In a possible implementation, in the case where the annotation information includes: the position information of the target object in the source domain sample image, the position information includes: the coordinate values ​​of the corner points of the detection box of the target object in the source domain sample image; the first transformation module, when performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain the corrected annotation information, is used to: determine the coordinate values ​​of the corner points of the detection box of the target object in the source domain corrected image based on the coordinate values ​​of the corner points of the detection box in the source domain original image and the correction transformation relationship information; and obtain the corrected annotation information based on the coordinate values ​​of the corner points of the detection box in the source domain corrected image.

[0031] In a possible implementation manner, when the training module uses the corrected image corresponding to the sample image to train the neural network to obtain the target neural network, the training module is used to: use the source domain corrected image, the target domain corrected image, and the corrected annotation information to train the neural network to obtain the target neural network; when the training module uses the source domain corrected image, the target domain corrected image, and the corrected annotation information to train the neural network to obtain the target neural network, the training module is used to: use the neural network to be trained to process the source domain corrected image and the target domain corrected image to obtain a first result corresponding to the source domain corrected image and a second result corresponding to the target domain corrected image; determine the model loss based on the first result, the second result, the corrected annotation information, and the weight values ​​corresponding to each first pixel point in the source domain corrected image; and use the model loss to train the neural network to obtain the target neural network.

[0032] In a possible implementation, the training module is further used to determine, for each first pixel in the source domain corrected image, a weight value of the first pixel based on a coordinate value of a second pixel corresponding to the first pixel in the source domain sample image, and a coordinate value of a second pixel corresponding to an adjacent first pixel of the first pixel in the source domain sample image.

[0033] In a fourth aspect, the embodiments of the present disclosure also provide an image processing device, comprising: an acquisition module, used to acquire an image to be processed; a second transformation module, used to perform a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed; a processing module, used to process the target corrected image using a target neural network generated by the neural network generation method as described in any one of the first aspects, to obtain an intermediate processing result of the target corrected image; and a third transformation module, used to obtain a target processing result of the image to be processed based on the intermediate processing result.

[0034] In a possible implementation, when the second transformation module performs a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed, the second transformation module is used to: determine a first target pixel point in the target corrected image obtained by performing a deformation correction transformation on the image to be processed; determine a second target pixel point in the image to be processed corresponding to the first target pixel point for each first target pixel point in the target corrected image and correction transformation relationship information between the image to be processed and the corresponding target corrected image; determine a pixel value of the first target pixel point based on the pixel value of the second target pixel point corresponding to the first target pixel point; and generate the target corrected image based on the pixel values ​​of each first target pixel point.

[0035] In a possible implementation, the second transformation module, when determining, based on the image to be processed, the first target pixel point in the target corrected image obtained by performing a deformation correction transformation on the image to be processed, is used to: determine the position information of the corner pixel points in the target corrected image in the target corrected image based on the position information of the corner pixel points in the image to be processed and the correction transformation relationship information; and determine the first target pixel point located in the target corrected image based on the position information of the corner pixel points in the target corrected image in the target corrected image.

[0036] In a possible implementation, when the target processing result of the image to be processed is obtained based on the intermediate processing result, the third transformation module is used to: perform an inverse deformation correction transformation on the intermediate processing result to obtain the target processing result of the image to be processed.

[0037] In a possible implementation, the target processing result includes: a semantic segmentation result of the image to be processed, and / or a target detection result of the image to be processed.

[0038] In a fifth aspect, an optional implementation of the present disclosure further provides an electronic device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps in the above-mentioned first aspect, or any possible implementation of the first aspect;

[0039] Or execute the steps in the above-mentioned second aspect, or any possible implementation manner of the second aspect.

[0040] In a sixth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the first aspect or any possible implementation of the first aspect are executed;

[0041] Or execute the steps in the above-mentioned second aspect, or any possible implementation manner of the second aspect.

[0042] For a description of the effects of the above-mentioned neural network generation device, image processing device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned neural network generation method and image processing method, which will not be repeated here.

[0043] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0045] Figure 1 A flowchart of a neural network training method provided by an embodiment of the present disclosure is shown;

[0046] Figure 2 A flowchart showing a specific method for performing deformation correction transformation on a sample image provided by an embodiment of the present disclosure is shown;

[0047] Figure 3 A schematic diagram showing the deformation of a target object provided by an embodiment of the present disclosure after being projected in an image coordinate system;

[0048] Figure 4 A schematic diagram of projection of a target object provided by an embodiment of the present disclosure after being projected in an image coordinate system and a spherical coordinate system respectively is shown;

[0049] Figure 5 An example of performing deformation correction transformation on a sample image provided by an embodiment of the present disclosure is shown;

[0050] Figure 6 An example of performing deformation correction transformation on annotation information provided by an embodiment of the present disclosure is shown;

[0051] Figure 7 An example of training a neural network to be trained provided by an embodiment of the present disclosure is shown;

[0052] Figure 8 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown;

[0053] Fig. 9 A schematic diagram of a neural network generation device provided by an embodiment of the present disclosure is shown;

[0054] Fig.10 A schematic diagram of an image processing device provided by an embodiment of the present disclosure is shown;

[0055] Fig.11 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0057] According to research, when using a neural network to perform target detection tasks or semantic segmentation tasks in an image, a large number of sample images with finely labeled information need to be used to train the neural network. However, it is laborious to obtain sample images. To solve this problem, unsupervised domain adaptation is currently usually used to train the neural network. When using unsupervised domain adaptation to train a neural network, it is necessary to use source domain images with labeled information and target domain images without labeled information to train the neural network. When training the neural network, the feature distributions of the source domain and the target domain are aligned by confusing the source domain image with the target domain image or by constraining the domain difference measurement to provide implicit or explicit supervision signals on the target domain. When aligning the feature distributions of the source domain and the target domain, domain differences caused by external factors such as background, lighting, and weather are usually taken into account. However, for the same target object, the external factors of the shooting are the same, but the cameras used are different. The imaging of the target object in images taken by different cameras will be deformed to different degrees, resulting in different imaging of the target object in images taken by different cameras, which in turn causes poor feature alignment between the source domain and the target domain, resulting in poor accuracy of the obtained neural network.

[0058] Based on the above research, the present disclosure provides a method for generating a neural network, which performs a corrective transformation on a sample image and then uses the obtained corrected image to train the neural network to be trained, thereby reducing the impact of the deformation of the target object in the image on the training results of the neural network and improving the accuracy of the target neural network.

[0059] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0060] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0061] To facilitate understanding of this embodiment, a method for generating a neural network disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the method for generating a neural network provided in an embodiment of the present disclosure is generally an electronic device with certain computing capabilities, and the electronic device includes, for example: a terminal device or a server or other processing device, and the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the method for generating a neural network may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0062] The following is an explanation of the method for generating a neural network provided in an embodiment of the present disclosure.

[0063] See also Figure 1 , which is a flow chart of a method for generating a neural network provided in an embodiment of the present disclosure, the method comprises steps S101 to S102, wherein:

[0064] S101: Perform deformation correction transformation on the sample image to obtain a corrected image corresponding to the sample image;

[0065] S102: Using the rectified image corresponding to the sample image to train the neural network to obtain a target neural network.

[0066] The disclosed embodiment performs a correction transformation on the sample image and then uses the obtained corrected image to train the neural network to be trained, thereby reducing the impact of the deformation of the target object in the image on the training result of the neural network and improving the accuracy of the target neural network.

[0067] The above S101 to S102 are described in detail below.

[0068] With respect to the above S101, all sample images may have annotation information. By performing a correction transformation on the sample images, the degree of deformation of the target object in the sample images can be reduced, thereby reducing the impact of the deformation of the target object on the training process of the neural network and improving the accuracy of the target neural network.

[0069] In another embodiment, when the sample image is used to perform supervised training on the neural network to be trained, the sample image also corresponds to annotation information; while the sample image is subjected to deformation correction transformation, the annotation information of the sample image can also be subjected to deformation correction transformation. The specific method of performing deformation correction transformation on the annotation information of the sample image is similar to the method of performing deformation correction transformation on the annotation information corresponding to the source domain sample image in the following embodiment to obtain the corrected annotation information, which will not be repeated here.

[0070] In another embodiment, the sample images may also include source domain sample images and target domain sample images.

[0071] The source domain sample images include multiple sample images with labeled information; the target domain sample images include multiple unlabeled sample images. The target neural network performs different image processing tasks, and the corresponding labeling information is different.

[0072] Exemplarily, if the target neural network is used to perform target detection tasks on images, the annotation information includes position information and category information of the target object in the source domain sample image; the position information is, for example, the coordinate values ​​of the pixel points corresponding to the vertices of the bounding box of the target object in the source domain sample image; the category information includes, for example: the category to which the target object belongs; if the target neural network is used to perform semantic segmentation processing on the image, the annotation information includes the semantic information of each pixel point in the source domain sample image.

[0073] In this way, a source domain corrected image and a target domain corrected image are obtained by performing deformation correction transformation on the source domain sample image and the target domain sample image respectively, and the annotation information corresponding to the source domain sample image is subjected to deformation correction transformation to obtain corrected annotation information. Then, the source domain corrected image, the target domain corrected image, and the corrected annotation information are used to train the neural network to be trained to obtain the target neural network, thereby reducing the influence of the deformation generated when the target object is projected onto the image on the feature alignment between the source domain and the target domain, thereby improving the accuracy of the neural network.

[0074] See also Figure 2 As shown, the embodiment of the present disclosure provides a specific method for performing deformation correction transformation on a sample image, including:

[0075] S201: Determine a first pixel point in a corrected image obtained by performing a deformation correction transformation on the sample image.

[0076] In a specific implementation, when determining the first pixel point in a corrected image obtained by performing a deformation correction transformation on the sample image based on the sample image, for example, the position information of the corner pixel points in the corrected image can be determined based on the position information of the corner pixel points in the sample image in the sample image and the correction transformation relationship information between the sample image and the corrected image; and the first pixel point located in the corrected image can be determined based on the position information of the corner pixel points in the corrected image in the corrected image.

[0077] When determining the first pixel point located in the corrected image, the corner pixel point in the corrected image can be used as the first pixel point at the outermost position in the corrected image, and then based on the position information of the corner pixel point in the corrected image, the value range of the coordinate value of the first pixel point on the u-axis and the value range on the v-axis are obtained, and the first pixel point located in the corrected image is determined by using the above-mentioned value range on the u-axis and the value range on the v-axis.

[0078] Since different sample images are acquired using cameras with different intrinsic parameters, such as different focal lengths, principal points, image sensor formats, etc., the target object will also produce different degrees of deformation in the images acquired by shooting the same target object at the same shooting angle. For cameras with different intrinsic parameters, the target object will produce different degrees of deformation in the imaged image.

[0079] For the same camera, different angles between the same target object and the camera optical axis will also cause different degrees of deformation of the projection of the target object in the image.

[0080] like Figure 3 A and Figure 3 As shown in Figure b, 15 identical target objects form a spherical array. All target objects are at the same distance from the camera, but are located at different positions relative to camera A. However, the imaging of the 15 target objects on the image plane will produce obvious deformation, and the deformation produced by target objects at different positions is different. Figure 3 As shown in c, it is Figure 3 It can be seen that after the image is transformed with deformation correction, the deformation degree of the target object is alleviated, which is beneficial to the subsequent feature alignment.

[0081] In order to correct the deformation of the target object when projected on the image plane, the example of the present disclosure provides a specific method for establishing correction transformation relationship information between a sample image and a corresponding correction image, and according to the correction transformation relationship information, the sample image is subjected to deformation correction transformation, thereby correcting the projection deformation of the target object to a certain extent. When training the neural network to be trained, the correction image is used to train the neural network to be trained, so as to reduce the influence of the projection deformation on the accuracy of the neural network.

[0082] Specifically, for example, the correction transformation relationship information may be generated in the following manner:

[0083] A rectangular coordinate system is constructed with the focus of the camera as the origin; the rectangular coordinate system includes x, y and z axes, wherein the z axis is the straight line where the optical axis of the camera is located.

[0084] And, establishing a spherical coordinate system with the optical center of the camera as the origin and the focal length of the camera as the radius; wherein the spherical coordinate system includes a u-axis and a v-axis;

[0085] Determine a first projection point of the target object in a rectangular coordinate system and a second projection point of the target object in a spherical coordinate system based on an incident angle of the target object relative to an optical axis of the camera;

[0086] Generate conversion relationship information between the first projection point and the second projection point based on a first coordinate value (X, Y) of the first projection point in the rectangular coordinate system and a second coordinate value (U, V) of the second projection point in the spherical coordinate system;

[0087] The conversion relationship information between the first projection point and the second projection point is used as the correction transformation relationship information.

[0088] In the specific implementation, Figure 4 As shown in a, it shows a specific example of the deformation of the target object in the image caused by the change in the angle between the target object and the camera optical axis, where l, m, and n are objects with similar structures. They have different relative positions with the camera optical axis, that is, different incident angles with respect to the camera optical axis, and the same distance from the camera optical center O. After being projected onto the camera imaging plane (that is, the rectangular coordinate system established based on the camera focus), the lengths of their images l', m', and n' are significantly different. Figure 4As can be seen in a, as the incident angle of the target object increases, its imaging width and imaging height increase, causing the projection of the target object at different positions on the image plane to be deformed to different degrees. This deformation will affect the feature alignment between the source domain sample image and the target domain sample image, resulting in poor accuracy of the neural network trained based on the source domain sample image and the target domain sample image. After projecting l, m, and n to the spherical coordinate system, their imaging l″, m″, and n″ have the same length.

[0089] Therefore, if Figure 4 As shown in b, the embodiment of the present disclosure establishes a rectangular coordinate system xFy with the focus F of the camera as the origin, and establishes a spherical coordinate system uv with the optical center of the camera as the origin and the focal length of the camera as the radius. For a target object located within the field of view of the camera, if the first coordinate value of the first projection point in the rectangular coordinate system is (X, Y), and the second coordinate value of the second projection point in the spherical coordinate system is (U, V), see Figure 4 Medium c and Figure 4 As shown in d, the first projection point and the second projection point satisfy the following formulas (1) to (3):

[0090]

[0091]

[0092] M′[U,V]=M[X,Y] (3)

[0093] Among them, (U, V) represents the coordinate value of the second projection point in the spherical coordinate system; (X(U), Y(V)) represents the coordinate value of the first projection point corresponding to the second projection point in the rectangular coordinate system.

[0094] f is the focal length of the camera, represents the incident angle α between the target object and the optical axis of the camera in the x-axis direction, represents the incident angle β between the target object and the optical axis of the camera in the y-axis direction. M[X,Y] represents the pixel value of the first projection point, and M′[U,V] represents the pixel value of the second projection point.

[0095] It can be seen from the above formulas (1) to (2) that the larger the incident angle α between the target object and the optical axis of the camera in the x-axis direction, the larger the deformation of the target object in the x-axis direction; the smaller the incident angle α, the smaller the deformation of the target object in the x-axis direction. Moreover, the larger the incident angle β between the target object and the optical axis of the camera in the y-axis direction, the larger the deformation of the target object in the y-axis direction; the smaller the incident angle β, the smaller the deformation of the target object in the y-axis direction.

[0096] The above formulas (1) to (2) are used as correction transformation relationship information.

[0097] When determining the position information of the corner pixel points in the corrected image based on the sample image, the position information of the corner pixel points in the sample image, and the correction transformation relationship information between the sample image and the corrected image, the corner pixel points in the sample image include, for example: P1 (X1, Y1), P2 (-X1, Y1), P3 (X1, -Y1) and P4 (-X1, -Y1),

[0098] According to the above formulas (1) and (2), the corner pixel points P1′(U1, V1), P2′(-U1, V1), P3′(U1, -V1), and P4′(-U1, -V1) in the corrected image can be obtained. Then, according to the corner pixel points in the corrected image, the coordinate value of the first pixel point in the corrected image in the corrected image has a value range of [-U1, U1] on the u-axis and a value range of [-V1, V1] on the v-axis. That is, the size of the corrected image obtained is 2U1×2V1. Based on the value range of the coordinate value of the first pixel point, the first pixel point in the corrected image can be determined.

[0099] S202: For each first pixel point in the corrected image and the correction transformation relationship information, determine a second pixel point corresponding to the first pixel point.

[0100] S203: Determine the pixel value of the first pixel point based on the pixel value of the second pixel point corresponding to the first pixel point.

[0101] S204: Generate the corrected image based on the pixel values ​​of each first pixel point.

[0102] Here, after each first pixel point is determined, the first coordinate value of each first pixel point in the corrected image is determined; then, according to the above formulas (1) and (2), the second coordinate value of the second pixel point corresponding to the first pixel point in the sample image is obtained, and then based on the second coordinate value of the second pixel point in the sample image, the corresponding second pixel point is determined.

[0103] After the second pixel point is determined, that is, the pixel value of the second pixel point can be determined, the pixel value of the second pixel point is determined as the pixel value of the first pixel point.

[0104] After the pixel values ​​of all the first pixel points are determined, a corrected image is obtained based on the pixel values ​​of the first pixel points and the coordinate values ​​of the first pixel points in the corrected sample image.

[0105] like Figure 5As shown, the embodiment of the present disclosure provides a specific example of obtaining a corrected image b after performing a deformation correction transformation on a sample image a.

[0106] In another embodiment of the present disclosure, when the sample image includes the above-mentioned source domain sample image and the target domain sample image, that is, both the source domain sample image and the target domain sample image can be obtained by Figure 2 The method shown in the figure is used to perform deformation correction transformation to obtain the corresponding corrected image. Figure 2 The embodiment shown performs a deformation correction transformation on the source domain sample image to obtain a source domain corrected image; Figure 2 The illustrated embodiment performs a deformation correction transformation on the target domain sample image to obtain a target domain corrected image.

[0107] In addition, since the annotation information of the source domain sample image is annotated on the basis of the source domain image, after the source domain sample image is deformed and corrected, in order to maintain the consistency between the source domain sample image and the annotation information, the annotation information corresponding to the source domain sample image must also be deformed and corrected to obtain the corrected annotation information.

[0108] Among them, when the annotation information is subjected to a deformation correction transformation, since the annotation information of the source domain sample image is related to the image processing task of the neural network, for example, if the image processing task of the neural network is semantic segmentation, the annotation information is the semantic information of each pixel in the source domain sample image, then a labeled image can be generated based on the semantic information of each pixel in the source domain sample image; a deformation correction transformation is performed on the labeled image to obtain a corrected labeled image of the labeled image; and based on the corrected labeled image, the corrected annotation information is obtained.

[0109] Here, the method of performing deformation correction transformation on the annotation image is similar to the method of performing deformation correction transformation on the sample image, and will not be described in detail here.

[0110] In addition, since there is a one-to-one correspondence between the pixels in the annotated image and the pixels in the source domain sample image, when performing a deformation correction transformation on the source domain sample image, in the process of determining the second pixel corresponding to the first pixel in the source domain corrected image, the semantic information of the second pixel can be used as the semantic information of the annotated pixel corresponding to the first pixel in the corrected annotated image to obtain the corrected annotated image.

[0111] For another example, if the processing task of the neural network is target detection, the annotation information of the source domain sample image includes: the position information of the target object in the source domain sample image, and the position information includes: the coordinate values ​​of the corner points of the detection box of the target object in the source domain sample image;

[0112] The step of performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain the corrected annotation information includes:

[0113] Determine the coordinate values ​​of the corner points of the detection frame of the target object in the source domain corrected image based on the coordinate values ​​of the corner points of the detection frame in the source domain original image and the correction transformation relationship information;

[0114] The correction annotation information is obtained based on the coordinate values ​​of the corner points of the detection box in the source domain corrected image.

[0115] In addition, in some cases, the position information of the target object in the source domain sample image can also be expressed as the coordinate value of the center point of the target object in the source domain sample image, and the offset of the corner point of the detection box of the target object relative to the center point of the target object. In this case, the coordinate value of the corner point of the detection box in the source domain sample image can be first obtained based on the coordinate value of the above-mentioned center point in the source domain sample image and the offset, and then the deformation correction transformation can be performed.

[0116] See also Figure 6 As shown, the embodiment of the present disclosure provides a specific example of obtaining a corrected annotated image c after performing a deformation correction transformation on an annotated image b of a source domain sample image a.

[0117] With respect to the above S102, when the corrected image corresponding to the sample image is used to train the neural network to be trained, for example, the sample image can be processed by the neural network to be trained, and then the model loss is obtained using the processing result, and the neural network to be trained is trained using the model loss to obtain the target neural network.

[0118] In another embodiment, the source domain rectified image, the target domain rectified image, and the rectified annotation information may be used to train the neural network to be trained to obtain a target neural network.

[0119] After using the source domain rectified image corresponding to the source domain sample image and the target domain rectified image corresponding to the target domain sample image and obtaining the correction annotation information, the source domain rectified image and the target domain rectified image can be processed respectively using the neural network to be trained to obtain a first result corresponding to the source domain rectified image and a second result corresponding to the target domain rectified image.

[0120] Then, based on the first result, the second result, the corrected annotation information, and the weight values ​​corresponding to each first pixel point in the source domain corrected image, the model loss is determined, and the neural network to be trained is trained using the model loss to obtain the target neural network.

[0121] Here, the model loss satisfies the following formula (4), for example:

[0122]

[0123] Among them, L task For example, the task loss for the image processing task to be performed by the neural network to be trained, such as the supervision loss; L da W represents the domain adaptation loss, such as the consistency loss between the source domain and the target domain determined based on the first result and the second result. R represents the weight matrix composed of the weight values ​​of the first pixel points in the source domain correction image. λ is a hyperparameter used to balance the weights of task loss and domain adaptation loss. R After weighting the model loss, the corrected annotation information obtained through the deformation correction transformation can be used to supervise the training process of the neural network to be trained, thereby speeding up the training process of the neural network to be trained.

[0124] The weight value corresponding to each first pixel point in the source domain rectified image may be determined, for example, in the following manner:

[0125] For each first pixel in the source domain corrected image, a weight value of the first pixel is determined based on a coordinate value of a second pixel corresponding to the first pixel in the source domain sample image and a coordinate value of a second pixel corresponding to an adjacent first pixel of the first pixel in the source domain sample image.

[0126] Here, when the sample image is subjected to a deformation correction transformation, the information of the first pixel point in the obtained corrected image is used to characterize the information of the second pixel point in an area of ​​the sample image, and as the distance between the first pixel point and the center of the spherical coordinate system uv corresponding to the corrected image increases, the area corresponding to the first pixel point in the sample image increases, that is, the corresponding area loses more information. Therefore, the weight value of the first pixel point is determined according to the coordinate value of the second pixel point corresponding to the adjacent first pixel point of the first pixel point in the source domain sample image, and as the distance between the first pixel point and the center of the spherical coordinate system uv corresponding to the corrected image increases, the weight value corresponding to the first pixel point increases, so that the neural network to be trained can learn more features of the area with more lost information, reduce the influence of the information loss caused by the deformation correction transformation in the sample image on the training process of the neural network to be trained, and improve the accuracy of the target neural network obtained.

[0127] For example, assume that the first pixel in the source domain rectified image is (U, V), and its weight value w R (U, V) satisfies the following formula (5):

[0128] w R (U,V)=(X(|U+1|)-X(|U|))×(Y(|V+1|)-Y(|V|)) (5)

[0129] in,

[0130] (|U+1|, |V+1|) represents the coordinate value of the adjacent first pixel point corresponding to the first pixel point (U, V).

[0131] Based on the above formula (5), the weight value of each first pixel in the source domain rectified image can be obtained.

[0132] After obtaining the weight value of each first pixel in the source domain rectified image, the model loss can be obtained based on the weight matrix composed of the weight values ​​of each first pixel, and then the neural network to be trained can be trained.

[0133] See also Figure 7 As shown, the embodiment of the present disclosure also provides a specific example of training a neural network to be trained, including:

[0134] Perform deformation correction transformation on the source domain image to obtain the source domain corrected image.

[0135] Perform deformation correction transformation on the target domain sample image to obtain the target domain corrected image.

[0136] Perform deformation correction transformation on the annotation information to obtain corrected annotation information.

[0137] The source domain rectified image is processed using the neural network to be trained to obtain a first result.

[0138] The target domain rectified image is processed using the neural network to be trained to obtain a second result.

[0139] The weight matrix is ​​used as supervision, the model loss is obtained based on the corrected annotation information, the first result and the second result, and the neural network to be trained is trained to obtain the neural network.

[0140] After the above training process, the obtained neural network has higher accuracy.

[0141] See also Figure 8 As shown, the embodiment of the present disclosure also provides an image processing method, including:

[0142] S801: Acquire the image to be processed;

[0143] S802: Performing deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed;

[0144] S803: Processing the target corrected image using a target neural network generated by the neural network generation method described in any embodiment of the present disclosure to obtain an intermediate processing result of the target corrected image;

[0145] S804: Based on the intermediate processing result, a target processing result of the image to be processed is obtained.

[0146] The disclosed embodiment obtains a target corrected image of the image to be processed by performing a deformation correction transformation on the image to be processed. Since the target neural network is trained by using sample images that have undergone deformation correction transformation, it has higher accuracy. Therefore, when the target corrected image is processed, the intermediate processing result obtained has higher accuracy; the target processing result of the image to be processed obtained based on the intermediate processing result also has higher accuracy.

[0147] Specifically, the intermediate processing result of the target corrected image is the result of processing the target corrected image using the target neural network.

[0148] Since the target corrected image has changed in size after the deformation correction transformation, the result of the target neural network processing the target corrected image cannot be directly used as the result of the image to be processed. Therefore, after obtaining the intermediate processing result of the target corrected image, it is necessary to obtain the target processing result of the image to be processed based on the intermediate processing result. The specific method of obtaining the target processing result of the image to be processed based on the intermediate processing result is described in the following embodiment, which will not be repeated here.

[0149] In a possible implementation manner, performing a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed includes:

[0150] Based on the image to be processed, determining a first target pixel point in a target corrected image obtained by performing a deformation correction transformation on the image to be processed;

[0151] For each first target pixel point in the target corrected image and the correction transformation relationship information between the image to be processed and the corresponding target corrected image, determine a second target pixel point in the image to be processed corresponding to the target pixel point;

[0152] Determine a pixel value of the first target pixel based on a pixel value of a second target pixel corresponding to the first target pixel;

[0153] The target corrected image is generated based on the pixel values ​​of each first target pixel point.

[0154] In a possible implementation manner, determining, based on the image to be processed, a first target pixel point in a target corrected image obtained by performing a deformation correction transformation on the image to be processed includes:

[0155] Determine the position information of the corner pixel points in the target corrected image in the target corrected image based on the position information of the corner pixel points in the target corrected image and the correction transformation relationship information;

[0156] Based on the position information of the corner pixel points in the target rectified image in the target rectified image, a first target pixel point located in the target rectified image is determined.

[0157] Here, the correction transformation relationship is obtained in the same way as above. Figure 1 The methods for obtaining the correction transformation relationship in the corresponding embodiments are similar and will not be described in detail here.

[0158] In a possible implementation manner, obtaining a target processing result of the image to be processed based on the intermediate processing result includes:

[0159] The intermediate processing result is subjected to an inverse deformation correction transformation to obtain a target processing result of the image to be processed.

[0160] Here, the inverse deformation correction transformation is performed on the intermediate processing result, for example, based on the above formulas (1) and (2). Given U, V, and f in formulas (1) and (2), the process of calculating X and Y is not described in detail.

[0161] In a possible implementation, the target processing result includes: a semantic segmentation result of the image to be processed, and / or a target detection result of the image to be processed.

[0162] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0163] Based on the same inventive concept, the embodiments of the present disclosure also provide a device for generating a neural network corresponding to the method for generating a neural network. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the method for generating a neural network in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0164] Reference Fig. 9 FIG. 1 is a schematic diagram of a neural network generation device provided by an embodiment of the present disclosure, wherein the device comprises:

[0165] A first transformation module 91 is used to perform a deformation correction transformation on the sample image to obtain a corrected image corresponding to the sample image;

[0166] The training module 92 is used to train the neural network to be trained using the rectified image corresponding to the sample image to obtain the target neural network.

[0167] In one possible implementation, the first transformation module 91, when performing a deformation correction transformation on a sample image, is used to: determine a first pixel point in a corrected image obtained by performing a deformation correction transformation on the sample image; determine a second pixel point in the sample image corresponding to the first pixel point for each first pixel point in the corrected image and correction transformation relationship information between the sample image and the corresponding corrected image; determine a pixel value of the first pixel point based on the pixel value of the second pixel point corresponding to the first pixel point; and generate the corrected image based on the pixel values ​​of each first pixel point.

[0168] In one possible implementation, the first transformation module 91, when determining the first pixel point in the corrected image obtained by performing a deformation correction transformation on the sample image, is used to: determine the position information of the corner pixel points in the corrected image based on the position information of the corner pixel points in the sample image and the correction transformation relationship information; and determine the first pixel point located in the corrected image based on the position information of the corner pixel points in the corrected image.

[0169] In a possible implementation, the invention further includes: a relationship determination module 93, configured to generate the correction transformation relationship information in the following manner:

[0170] Construct a rectangular coordinate system with the focus of the camera as the origin; and establish a spherical coordinate system with the optical center of the camera as the origin and the focal length of the camera as the radius;

[0171] Determine a first projection point of the target object in a rectangular coordinate system and a second projection point of the target object in a spherical coordinate system based on an incident angle of the target object relative to an optical axis of the camera;

[0172] Generate conversion relationship information between the first projection point and the second projection point based on a first coordinate value of the first projection point in the rectangular coordinate system and a second coordinate value of the second projection point in the spherical coordinate system;

[0173] The conversion relationship information between the first projection point and the second projection point is used as the correction transformation relationship information.

[0174] In a possible implementation manner, the sample image includes a source domain sample image and a target domain sample image; and the first transformation module 91, when performing a deformation correction transformation on the sample image to obtain a corrected image corresponding to the sample image, is used to:

[0175] The source domain sample image and the target domain sample image are respectively subjected to deformation correction transformation to obtain the source domain corrected image and the target domain corrected image.

[0176] In a possible implementation manner, the first transformation module 91 is further configured to perform a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain corrected annotation information.

[0177] In a possible implementation manner, in the case where the annotation information includes: semantic information of each pixel in the source domain sample image, the first transformation module 91, when performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain the corrected annotation information, is used to:

[0178] Generate a labeled image based on the semantic information of each pixel in the source domain sample image;

[0179] Performing a deformation correction transformation on the annotated image to obtain a corrected annotated image of the annotated image;

[0180] The corrected annotation information is obtained based on the corrected annotation image.

[0181] In a possible implementation, in the case where the annotation information includes: position information of the target object in the source domain sample image, the position information includes: coordinate values ​​of corner points of a detection box of the target object in the source domain sample image;

[0182] The first transformation module 91, when performing deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain the corrected annotation information, is used to:

[0183] Determine the coordinate values ​​of the corner points of the detection frame of the target object in the source domain corrected image based on the coordinate values ​​of the corner points of the detection frame in the source domain original image and the correction transformation relationship information;

[0184] The correction annotation information is obtained based on the coordinate values ​​of the corner points of the detection box in the source domain corrected image.

[0185] In a possible implementation manner, the training module 92, when using the corrected image corresponding to the sample image to train the neural network to be trained to obtain the target neural network, is used to: use the source domain corrected image, the target domain corrected image, and the corrected annotation information to train the neural network to be trained to obtain the target neural network; the training module 92, when using the source domain corrected image, the target domain corrected image, and the corrected annotation information to train the neural network to be trained to obtain the target neural network, is used to: use the neural network to be trained to process the source domain corrected image and the target domain corrected image to obtain a first result corresponding to the source domain corrected image and a second result corresponding to the target domain corrected image; determine the model loss based on the first result, the second result, the corrected annotation information, and the weight values ​​corresponding to each first pixel point in the source domain corrected image; and use the model loss to train the neural network to be trained to obtain the target neural network.

[0186] In a possible implementation manner, the training module 92 is further used to:

[0187] For each first pixel in the source domain corrected image, a weight value of the first pixel is determined based on a coordinate value of a second pixel corresponding to the first pixel in the source domain sample image and a coordinate value of a second pixel corresponding to an adjacent first pixel of the first pixel in the source domain sample image.

[0188] For the description of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant description in the above-mentioned embodiment of the method for generating a neural network, which will not be described in detail here.

[0189] See also Fig.10 As shown, the embodiment of the present disclosure also provides an image processing device, including:

[0190] An acquisition module 101 is used to acquire an image to be processed;

[0191] A second transformation module 102 is used to perform a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed;

[0192] A processing module 103, configured to process the target corrected image using a target neural network generated by the neural network generation method described in any embodiment of the present disclosure, to obtain an intermediate processing result of the target corrected image;

[0193] The third transformation module 104 is used to obtain a target processing result of the image to be processed based on the intermediate processing result.

[0194] In a possible implementation manner, the second transformation module 102, when performing deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed, is configured to:

[0195] Based on the image to be processed, determining a first target pixel point in a target corrected image obtained by performing a deformation correction transformation on the image to be processed;

[0196] For each first target pixel point in the target corrected image and the correction transformation relationship information between the image to be processed and the corresponding target corrected image, determine a second target pixel point in the image to be processed corresponding to the first target pixel point;

[0197] Determine a pixel value of the first target pixel based on a pixel value of a second target pixel corresponding to the first target pixel;

[0198] The target corrected image is generated based on the pixel values ​​of each first target pixel point.

[0199] In a possible implementation manner, when determining, based on the image to be processed, a first target pixel point in a target corrected image obtained by performing a deformation correction transformation on the image to be processed, the second transformation module 102 is configured to:

[0200] Determine the position information of the corner pixel points in the target corrected image in the target corrected image based on the position information of the corner pixel points in the target corrected image and the correction transformation relationship information;

[0201] Based on the position information of the corner pixel points in the target rectified image in the target rectified image, a first target pixel point located in the target rectified image is determined.

[0202] In a possible implementation manner, when obtaining the target processing result of the image to be processed based on the intermediate processing result, the third transformation module 104 is configured to:

[0203] The intermediate processing result is subjected to an inverse deformation correction transformation to obtain a target processing result of the image to be processed.

[0204] In a possible implementation, the target processing result includes: a semantic segmentation result of the image to be processed, and / or a target detection result of the image to be processed.

[0205] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above-mentioned image processing method embodiment, which will not be described in detail here.

[0206] The present disclosure also provides an electronic device, such as Fig.11 FIG. 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, including:

[0207] A processor 111 and a memory 112; the memory 112 stores machine-readable instructions executable by the processor 111, and the processor 111 is used to execute the machine-readable instructions stored in the memory 112. When the machine-readable instructions are executed by the processor 111, the processor 111 performs the following steps:

[0208] The sample image is subjected to deformation correction transformation to obtain a corrected image corresponding to the sample image; the corrected image corresponding to the sample image is used to train the neural network to be trained to obtain a target neural network.

[0209] Or follow these steps:

[0210] Get the image to be processed;

[0211] Performing a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed;

[0212] Processing the target corrected image using a target neural network generated by the neural network generation method described in any embodiment of the present disclosure to obtain an intermediate processing result of the target corrected image;

[0213] Based on the intermediate processing result, a target processing result of the image to be processed is obtained.

[0214] The above-mentioned memory 112 includes internal memory 1121 and external memory 1122; the memory 1121 here is also called internal memory, which is used to temporarily store the calculation data in the processor 111, as well as the data exchanged with the external memory 1122 such as the hard disk. The processor 111 exchanges data with the external memory 1122 through the internal memory 1121.

[0215] The specific execution process of the above instructions can refer to the steps of the neural network generation method or image processing method described in the embodiments of the present disclosure, which will not be repeated here.

[0216] The embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for generating a neural network described in the above method embodiment or the steps of the image processing method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0217] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the neural network generation method or the image processing method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0218] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0219] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0220] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0222] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0223] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for generating a neural network, characterized in that: include: Based on the preset correction transformation relationship information between the sample image and the corresponding correction image, the sample image is subjected to deformation correction transformation to obtain the correction image corresponding to the sample image; the correction transformation relationship is used to correct the projection deformation of the object in the sample image; the sample image includes a source domain sample image and a target domain sample image; the source domain sample image includes a plurality of sample images with annotation information; the target domain sample image includes a plurality of unannotated sample images; Using the corrected image corresponding to the sample image to train the neural network to obtain a target neural network; the target neural network is used to perform target detection or semantic segmentation on the image; The correction transformation relationship information is generated by the following steps: Construct a rectangular coordinate system with the focus of the camera as the origin; and establish a spherical coordinate system with the optical center of the camera as the origin and the focal length of the camera as the radius; Determine a first projection point of the target object in a rectangular coordinate system and a second projection point of the target object in a spherical coordinate system based on an incident angle of the target object relative to an optical axis of the camera; Based on a first coordinate value of the first projection point in the rectangular coordinate system and a second coordinate value of the second projection point in the spherical coordinate system, generating conversion relationship information between the first projection point and the second projection point as the correction transformation relationship information; The method of performing a deformation correction transformation on the sample image based on the preset correction transformation relationship information between the sample image and the corresponding correction image to obtain a correction image corresponding to the sample image includes: Determining the position information of the corner pixel points in the corrected image in the corrected image based on the position information of the corner pixel points in the sample image and the correction transformation relationship information; And according to the pixel points within the range enclosed by the position information of the corner pixel points in the corrected image, determine the first pixel point in the corrected image; wherein, the coordinate values ​​of the corner pixel points P1′(U1, V1), P2′(-U1, V1), P3′(U1, -V1), P4′(-U1, -V1), the coordinate values ​​of the first pixel point in the corrected image in the corrected image have a value range of [-U1, U1] on the u-axis and a value range of [-V1, V1] on the v-axis, and the size of the corrected image is: 2U1×2V1; For each first pixel point in the corrected image and the correction transformation relationship information between the sample image and the corresponding corrected image, determine a second pixel point in the sample image corresponding to the first pixel point; Determine a pixel value of the first pixel based on a pixel value of a second pixel corresponding to the first pixel; The corrected image is generated based on the pixel values ​​of each first pixel point.

2. The generation method according to claim 1, characterized in that: The step of performing deformation correction transformation on the sample image to obtain a corrected image corresponding to the sample image includes: The source domain sample image and the target domain sample image are respectively subjected to deformation correction transformation to obtain the source domain corrected image and the target domain corrected image.

3. The generation method according to claim 2, characterized in that: The method further comprises: Perform a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain corrected annotation information.

4. The generation method according to claim 3, characterized in that: In the case where the annotation information includes semantic information of each pixel in the source domain sample image, The step of performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain the corrected annotation information includes: Generate a labeled image based on the semantic information of each pixel in the source domain sample image; Performing a deformation correction transformation on the annotated image to obtain a corrected annotated image of the annotated image; The corrected annotation information is obtained based on the corrected annotation image.

5. The generation method according to claim 3, characterized in that: In the case where the annotation information includes the position information of the target object in the source domain sample image, the position information includes: the coordinate values ​​of the corner points of the detection box of the target object in the source domain sample image; The step of performing a deformation correction transformation on the annotation information corresponding to the source domain sample image to obtain the corrected annotation information includes: Determine the coordinate values ​​of the corner points of the detection frame of the target object in the source domain rectified image based on the coordinate values ​​of the corner points of the detection frame in the source domain sample image and the correction transformation relationship information; The correction annotation information is obtained based on the coordinate values ​​of the corner points of the detection box in the source domain corrected image.

6. The generation method according to claim 4, characterized in that: The method of training the neural network to be trained using the rectified image corresponding to the sample image to obtain the target neural network includes: Using the source domain rectified image, the target domain rectified image, and the rectified annotation information, training the neural network to be trained to obtain a target neural network; The step of training the neural network to be trained by using the source domain rectified image, the target domain rectified image, and the rectified annotation information to obtain a target neural network includes: Using the neural network to be trained to process the source domain rectified image and the target domain rectified image, to obtain a first result corresponding to the source domain rectified image and a second result corresponding to the target domain rectified image; A model loss is determined based on the first result, the second result, the correction annotation information, and the weight value corresponding to each first pixel point in the source domain corrected image; wherein the model loss satisfies the following formula: ;in, is the task loss for the image processing task to be performed by the neural network to be trained, represents the consistency loss between the source domain and the target domain determined based on the first result and the second result, represents the weight matrix composed of the weight values ​​of each first pixel in the source domain correction image, is a hyperparameter used to balance the weight of task loss and domain adaptation loss; The neural network to be trained is trained using the model loss to obtain the target neural network.

7. The generation method according to claim 6, characterized in that: Also includes: For each first pixel in the source domain corrected image, a weight value of the first pixel is determined based on a coordinate value of a second pixel corresponding to the first pixel in the source domain sample image and a coordinate value of a second pixel corresponding to an adjacent first pixel of the first pixel in the source domain sample image.

8. An image processing method, characterized in that: include: Get the image to be processed; Performing a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed; Processing the target corrected image using a target neural network generated by the neural network generation method according to any one of claims 1 to 7 to obtain an intermediate processing result of the target corrected image; An inverse deformation correction transformation is performed on the intermediate processing result to obtain a target processing result of the image to be processed, wherein the inverse deformation correction transformation is an inverse process of the deformation correction transformation.

9. The image processing method according to claim 8, characterized in that: The step of performing a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed includes: Determine the position information of the corner pixel points in the target corrected image in the target corrected image based on the position information of the corner pixel points in the target corrected image and the correction transformation relationship information; Determine the pixel points within the range enclosed by the position information of the corner pixel points in the target corrected image as the first target pixel points in the target corrected image; wherein, the coordinate values ​​of the first pixel point in the target corrected image obtained by the corner pixel points P1′(U1, V1), P2′(-U1, V1), P3′(U1, -V1), and P4′(-U1, -V1) on the u-axis are in the range of [-U1, U1], and on the v-axis are in the range of [-V1, V1], and the size of the target corrected image is: 2U1×2V1; For each first target pixel point in the target corrected image and the correction transformation relationship information between the image to be processed and the corresponding target corrected image, determine a second target pixel point in the image to be processed corresponding to the first target pixel point; Determine a pixel value of the first target pixel based on a pixel value of a second target pixel corresponding to the first target pixel; The target corrected image is generated based on the pixel values ​​of each first target pixel point.

10. The image processing method according to claim 8 or 9, characterized in that: The target processing result includes: a semantic segmentation result of the image to be processed, and / or a target detection result of the image to be processed.

11. A device for generating a neural network, characterized in that: include: A first transformation module is used to perform a deformation correction transformation on the sample image based on the correction transformation relationship information between the preset sample image and the corresponding correction image to obtain a correction image corresponding to the sample image; the correction transformation relationship is used to correct the projection deformation of the object in the sample image; the sample image includes a source domain sample image and a target domain sample image; the source domain sample image includes a plurality of sample images with annotation information; the target domain sample image includes a plurality of unannotated sample images; A training module, used to train the neural network to be trained using the rectified image corresponding to the sample image to obtain a target neural network; the target neural network is used to perform target detection or semantic segmentation on the image; The relationship determination module is used to generate the correction transformation relationship information in the following manner: Construct a rectangular coordinate system with the focus of the camera as the origin; and establish a spherical coordinate system with the optical center of the camera as the origin and the focal length of the camera as the radius; Determine a first projection point of the target object in a rectangular coordinate system and a second projection point of the target object in a spherical coordinate system based on an incident angle of the target object relative to an optical axis of the camera; Based on a first coordinate value of the first projection point in the rectangular coordinate system and a second coordinate value of the second projection point in the spherical coordinate system, generating conversion relationship information between the first projection point and the second projection point as the correction transformation relationship information; The method performs a deformation correction transformation on the sample image based on the correction transformation relationship information between the preset sample image and the corresponding corrected image to obtain a corrected image corresponding to the sample image, including: determining the position information of the corner pixel points in the corrected image in the corrected image based on the position information of the corner pixel points in the sample image and the correction transformation relationship information; and determining the first pixel point in the corrected image according to the pixel points within the range surrounded by the position information of the corner pixel points in the corrected image in the corrected image; wherein the corner pixel points P1′(U1, V1), P2 ′(-U1, V1), P3′(U1, -V1), P4′(-U1, -V1), the coordinate value of the first pixel point in the corrected image in the corrected image has a value range of [-U1, U1] on the u-axis and a value range of [-V1, V1] on the v-axis, and the size of the corrected image is: 2U1×2V1; for each first pixel point in the corrected image, and the correction transformation relationship information between the sample image and the corresponding corrected image, determine the second pixel point in the sample image corresponding to the first pixel point; based on the pixel value of the second pixel point corresponding to the first pixel point, determine the pixel value of the first pixel point; based on the pixel values ​​of each first pixel point, generate the corrected image.

12. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A second transformation module is used to perform a deformation correction transformation on the image to be processed to obtain a target corrected image of the image to be processed; A processing module, configured to process the target corrected image using a target neural network generated by the neural network generation method according to any one of claims 1 to 7, to obtain an intermediate processing result of the target corrected image; The third transformation module is used to perform an inverse deformation correction transformation on the intermediate processing result to obtain a target processing result of the image to be processed, wherein the inverse deformation correction transformation is an inverse process of the deformation correction transformation.

13. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the method for generating a neural network as described in any one of claims 1 to 7, or executes the image processing method as described in any one of claims 8 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by an electronic device, the electronic device executes the method for generating a neural network as described in any one of claims 1 to 7, or executes the image processing method as described in any one of claims 8 to 10.

Images