Training method of image processing network, image processing method and device

Abstract

The embodiment of the application discloses a kind of training method of image processing network, image processing method and device, wherein the method comprises: determining reference pixel based on the training image of annotation true value;With the reference pixel as starting point, the probability of cutting when the training image is processed by image processing network based on the Markov chain of the training image is determined;Based on the output result that the image processing network processes training cutting region and the true value, the network parameter value of the image processing network and the cutting probability are adjusted, and the image processing network trained is obtained;The training cutting region is obtained after the training image is cut based on the cutting probability.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer vision technology, and in particular to a training method for an image processing network, an image processing method, and an apparatus. Background Technology

[0002] Channel pruning has been widely used for model acceleration and compression to deploy parameterized convolutional neural networks (CNNs) on embedded or mobile devices. One related technique involves channel pruning of the network using Differentiable Markov Channel Pruning (DMCP). In DMCP, the channel pruning process is modeled as a Markov chain to reduce the search space. However, in gaze estimation, the inclusion of redundant pixels in the training images leads to low accuracy in gaze estimation using the trained CNN. Summary of the Invention

[0003] In view of this, embodiments of this application provide at least one training method for an image processing network, an image processing method, and an apparatus.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a method for training an image processing network, the method comprising:

[0006] Based on the labeled ground truth training image, determine the reference pixel;

[0007] Starting from the reference pixel, the cropping probability of the image processing network when processing the training image is determined based on the Markov chain of the training image.

[0008] Based on the output of the image processing network processing the training cropping region and the ground truth, the network parameter values ​​of the image processing network and the cropping probability are adjusted to obtain the trained image processing network; the training cropping region is obtained by cropping the training image based on the cropping probability.

[0009] Secondly, embodiments of this application provide an image processing method, the method comprising:

[0010] Obtain the image to be processed;

[0011] The image to be processed is cropped pixel by pixel based on the cropping probability of the trained image processing network to obtain the cropping region to be processed; wherein the trained image processing network is trained based on the method of the first aspect described above;

[0012] The trained image processing network is used to process the cropped region to obtain the processing result of the image.

[0013] Thirdly, embodiments of this application provide a training apparatus for an image processing network, comprising:

[0014] The first determination module is used to determine reference pixels based on the training image with labeled ground truth values;

[0015] The second determining module is used to determine the cropping probability when the image processing network processes the training image, starting from the reference pixel and based on the Markov chain of the training image.

[0016] The first adjustment module is used to adjust the network parameter values ​​and the cropping probability of the image processing network based on the output result of the image processing network processing the training cropping region and the ground truth value, so as to obtain the trained image processing network; the training cropping region is obtained by cropping the training image based on the cropping probability.

[0017] Fourthly, embodiments of this application provide an image processing apparatus, comprising:

[0018] The first acquisition module is used to acquire the image to be processed;

[0019] The first cropping module is used to crop pixels of the image to be processed based on the cropping probability of the trained image processing network to obtain the cropping region to be processed; wherein the trained image processing network is trained based on the training method of the above-mentioned image processing network.

[0020] The first processing module is used to process the cropped region to be processed using the trained image processing network to obtain the processing result of the image to be processed.

[0021] Fifthly, embodiments of this application provide a computer device including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the first or second aspect described above.

[0022] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the first or second aspect described above.

[0023] This application provides a computer program including computer-readable code. When the computer-readable code is run in a computer device, the processor in the computer device performs some or all of the steps in the first or second aspect described above.

[0024] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above method.

[0025] In this embodiment, a reference pixel is determined in the training image labeled with ground truth, facilitating the selection of image regions to be retained in the subsequent training image. Then, using this reference pixel as a starting point, the cropping operation on the training image is modeled as a Markov process combined with the reference pixel as the starting point, enabling the analysis of the cropping probability of each pixel in the training image. Subsequently, the network parameters and the cropping probability of each pixel in the image processing network are optimized using the predicted output of the training image and the ground truth value of the training image, resulting in a trained image processing network. Thus, by introducing a Markov chain to estimate the cropping probability of the training image, and by adjusting the cropping probability and network parameter values ​​using the output of the training image and the ground truth value, the cropping probability and network parameter values ​​can be optimized, thereby improving the performance of the trained image processing network.

[0026] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0028] Figure 1 A schematic diagram illustrating the implementation flow of a training method for an image processing network provided in an embodiment of this application;

[0029] Figure 2 This is a schematic diagram illustrating another implementation flow of a training method for an image processing network provided in an embodiment of this application;

[0030] Figure 3 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0031] Figure 4 A schematic diagram of the composition structure of a Markov chain provided in an embodiment of this application;

[0032] Figure 5 A schematic diagram illustrating an application scenario of a training method for an image processing network provided in this application embodiment;

[0033] Figure 6This is a schematic diagram illustrating an application scenario of an image processing method provided in an embodiment of this application;

[0034] Figure 7 This is a schematic diagram illustrating another application scenario of the training method for the image processing network provided in the embodiments of this application;

[0035] Figure 8A A schematic diagram illustrating the structural composition of a training device for an image processing network provided in an embodiment of this application;

[0036] Figure 8B This is a schematic diagram of the composition structure of an image processing device provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0041] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0042] 1) Computer vision refers to machine vision that uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs graphic processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection.

[0043] 2) Model compression aims to reduce the computational load, the number / volume of model parameters, and the inference time of the model.

[0044] 3) Markov Chain: A Markov chain is a set of discrete random variables. For example, given a set of random variables, if the values ​​of the random variables are all within a countable set, it is called a Markov chain. The countable set is called the state space, and the values ​​of the Markov chain within the state space are called states. The Markov property, also known as "memorylessness," means that the random variable at step t+1 is conditionally independent of the remaining random variables after the random variable at step t is given. In this embodiment, the cropping operation on the input image is modeled as a Markov chain, so that the cropping probability of the next pixel depends on the cropping probability of the previous pixel.

[0045] This application provides a training method for an image processing network, which can be executed by a processor of a computer device. The computer device can refer to a server, laptop computer, tablet computer, desktop computer, smart TV, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or any other device with data processing capabilities. Figure 1 This is a schematic diagram illustrating the implementation flow of a training method for an image processing network provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps S101 to S103:

[0046] Step S101: Determine reference pixels based on the training images labeled with ground truth.

[0047] In some embodiments, the training image can be a sample image with labeled ground truth values. The scene of the training image can be selected based on the task of the image processing network to be trained. The training image can be an image with complex content or an image with simple content. In some possible implementations, if the task of the image processing network is gaze estimation, then the training image can be an image including the eyes from various viewpoints, where the labeled ground truth values ​​are the gazes of the eyes. If the task of the image processing network is eye detection, then the training image can be an image including the face, where the labeled ground truth values ​​are the eyes in the face. If the task of the image processing network is vehicle recognition, then the training image is an image of a traffic scene, where the labeled ground truth values ​​are the vehicles in the image.

[0048] In some embodiments, the reference pixel can be a pixel selected in any frame of the training image. This reference pixel is used to characterize the center position of the image region to be retained during pixel cropping. Thus, the reference pixel can be the center pixel of the training image, or a pixel at a preset position in the training image. This preset position can be a user-defined position, or a position within a certain range in the training image, such as the image region containing a circle with a radius one-quarter of the image width. During network training, for each frame of the training image, a reference pixel is selected as the center pixel of the retained image region. Therefore, by selecting a reference pixel in the training image, it is easier to select the image region to be retained in subsequent training images.

[0049] Step S102: Starting from the reference pixel, determine the cropping probability when the image processing network processes the training image based on the Markov chain of the training image.

[0050] In some embodiments, for any training image frame, starting with a reference pixel in the training image, the cropping probability of a pixel in the training image is determined according to the Markov chain of the training image. The Markov chain of the training image can be a Markov chain that models the cropping process of the training image. In this Markov chain, the states correspond to pixels being retained, and the transition probabilities between two adjacent states correspond to the retention probability of the next pixel when the current pixel is retained. The retention probability of each state can be calculated by multiplying the transition probabilities; this retention probability is considered as the importance of the pixel.

[0051] In some possible implementations, during network training, starting with a reference pixel in the training image, the cropping probability of the next pixel adjacent to that reference pixel in the training image can be determined using the cropping probability of that reference pixel in the Markov chain. This allows for the determination of the cropping probability of each pixel in the training image. Thus, by modeling the cropping operation on the training image as a Markov process, the cropping probability of each pixel in the training image, starting from the reference pixel, can be analyzed.

[0052] Step S103: Based on the output result of the image processing network processing the training cropping region and the ground truth, adjust the network parameter values ​​and the cropping probability of the image processing network to obtain the trained image processing network.

[0053] In some embodiments, the output is related to the function implemented by the image processing network. If the function implemented by the image processing network is gaze estimation, then the output is a prediction result of the gaze estimation in the training image by the image processing network. Thus, by using this output and the labeled ground truth, the loss of the image processing network can be determined, and then the network parameter values ​​and the cropping probability in the image processing network can be adjusted using this loss, so that the loss output of the trained image processing network converges.

[0054] In some possible implementations, the network parameter values ​​of the image processing network include at least the following: the weights may also include channels to be pruned. During network training, at least the weights and pruning probabilities are optimized to obtain a trained image processing network that includes adjusted pruning probabilities and adjusted network parameter values.

[0055] In some embodiments, the training cropping region is obtained by cropping the training image based on the cropping probability. During network training, after determining the cropping probability of pixels through a Markov chain, the training image is cropped using this probability, thereby utilizing image processing to process the cropping region of the network to optimize the image processing network. In a specific example, taking the function of this image processing network as gaze estimation, the output result is the gaze estimation result, which can be achieved through the following steps:

[0056] The first step is to determine the training cropping region of the training image based on the cropping probability of the image processing network.

[0057] In some embodiments, the training image is cropped using the cropping probability corresponding to each pixel to obtain the training cropping region of the training image. In some possible implementations, the cropping probabilities of multiple pixels in the training image are represented as vectors, and these vectors are multiplied by the vector representing each pixel in the image. The result of the multiplication is the training cropping region of the training image.

[0058] The second step involves using the image processing network to estimate the gaze distance of the training cropped region, thereby obtaining the output result.

[0059] In some embodiments, the image processing network is a network for gaze estimation. In this case, the cropped training image, i.e., the training cropped region of the training image, is input into the image processing network to perform gaze estimation and obtain the output result. In this way, by applying the cropped training image during the training process, the transition probabilities between each pixel in the Markov chain can be optimized.

[0060] In this embodiment, a reference pixel is determined in the training image labeled with ground truth, facilitating the selection of image regions to be retained in the subsequent training image. Then, using this reference pixel as a starting point, the cropping operation on the training image is modeled as a Markov process combined with the reference pixel as the starting point, enabling the analysis of the cropping probability of each pixel in the training image. Finally, the network parameter values ​​and the cropping probability of each pixel in the image processing network are optimized using the predicted output of the training image and the ground truth value of the training image, resulting in a fully trained image processing network. Thus, by introducing a Markov chain to estimate the cropping probability of the training image, and by adjusting the cropping probability and network parameter values ​​using the output of the training image and the ground truth value, the image processing performance of the image processing network can be optimized.

[0061] In some embodiments, the image processing network can be a neural network for gaze estimation, where the training image is a facial image annotated with gaze information, i.e., the ground truth image is the annotated gaze information of the eyes in the face. Thus, the output corresponding to the training image includes the prediction result obtained by the image processing network performing gaze estimation on the training image. Therefore, by training the image processing network using training images annotated with gaze information, and by introducing a Markov chain to crop the pixels of the training image during implementation, and training the network using the cropped image, the accuracy of gaze estimation by the trained image processing network can be improved.

[0062] In some possible implementations, in the context of gaze estimation, the labeled gaze information and the gaze information in the output of the training image include at least one of the following: the pitch angle of the gaze, the yaw angle of the gaze, and the roll angle of the gaze. Thus, by incorporating gazes from various perspectives into the labeled gaze information, the types of gazes in the training image can be enriched, enabling the trained image processing network to accurately predict gaze angles from various perspectives.

[0063] In some embodiments, by using the center pixel of the training image as a reference pixel, the cropping probability of each pixel in the training image can be determined starting from that center pixel. That is, the above step S101 can be implemented by the following step S111 (not shown in the figure):

[0064] Step S111: Determine the center pixel in the training image as the reference pixel.

[0065] Here, during the training process of the image processing network, for each batch of training images input into the network, the pixel located at the center of that training image is determined, i.e., the center pixel. During image acquisition, the region of interest will be close to the center of the image, so the center pixel identified in the training image is very likely to be contained within the region of interest.

[0066] Here, the center pixel is used as the reference pixel, and the cropping probability of each pixel is determined starting from the center pixel. Since the probability that the center pixel is contained in the region of interest is relatively high, the probability that the cropped region finally cropped starting from the center pixel contains the region of interest is relatively high.

[0067] Step S111 above uses the center pixel in the training image as the reference pixel, which makes it more likely that the cropped region based on the center pixel will include the region of interest.

[0068] After setting the reference pixel through step S111 above, step S102 above can be achieved through the following step S112 (not shown in the figure):

[0069] Step S112: Starting from the center pixel, determine the cropping probability of each pixel in the training image based on the Markov chain.

[0070] Here, in the training image, starting from the center pixel, the cropping probability of the next pixel can be determined according to the cropping probability of the center pixel in the Markov chain, and thus the cropping probability of each pixel in the training image can be obtained.

[0071] In some possible implementations, the clipping probability of the center pixel in the Markov chain is determined by multiplying the transition probability from the center pixel to the next pixel by multiplying the transition probabilities of all pixels before the next pixel.

[0072] In this embodiment, by taking the center pixel of the training image as the starting point and combining the cropping probability of that center pixel in the Markov chain, the cropping probability of other pixels can be obtained quickly and accurately.

[0073] In some embodiments, during the process of determining the cropping probability of each pixel, the transition probability from the center pixel to other pixels is obtained through a Markov chain, and the cropping probability of each pixel can be obtained through this transition probability. That is, the above step S113 can be implemented through the following steps:

[0074] The first step is to determine the transition probability from the center pixel to the next pixel in the Markov chain.

[0075] In some embodiments, since each state in the Markov chain represents the pixel corresponding to that state being retained, it is possible to obtain the transition probability from the center pixel to the next retained pixel.

[0076] The second step is to determine the cropping probability of the next pixel based on the transition probability of the next pixel and the transition probabilities of the multiple pixels preceding the next pixel.

[0077] In some embodiments, the transition probability is multiplied by the transition probabilities of all pixels preceding the next pixel to obtain the cropping probability of the next pixel. Thus, by obtaining the transition probabilities from the center pixel to the next pixel to be retained through a Markov chain, and multiplying these transition probabilities to obtain the cropping probability of the next pixel, the training image can be cropped using this cropping probability to optimize the training image.

[0078] In some embodiments, the cropping probability of a pixel can be determined in the following two ways:

[0079] Method 1: Based on a Markov chain in at least one direction originating from the center pixel, the clipping probability of each pixel in at least one direction originating from the center pixel is set isotropically.

[0080] Here, for at least one direction originating from the center pixel, the cropping probability is set according to the isotropic nature of the Markov chain; thus, the cropping probability of each pixel in multiple directions originating from the center pixel is the same. This at least one direction includes directions to the left, right, up, or down from the center pixel; it may also include directions with a certain angle between them in the horizontal or vertical direction. For example, with the center pixel as the origin, this at least one direction can be the positive or negative direction of the X-axis where the origin is located. In this way, in the training image, for each pixel in at least one direction originating from the center pixel, the cropping probability of that pixel is obtained through the transition probabilities between pixels in the Markov chain, accurately determining whether a pixel in each direction needs to be retained, and ensuring that the probability of a pixel being retained is the same in each direction.

[0081] Method 2: Based on the Markov chain in the symmetrical propagation direction starting from the center pixel, set the clipping probability of the pixels in the symmetrical propagation direction starting from the center pixel.

[0082] Here, in the training image, the center pixel is used as a symmetry point. Based on a Markov chain that propagates symmetrically from the center pixel, the cropping probability of the corresponding pixel is determined. That is, the cropping probability of the pixel is set according to the Markov chain of left-right symmetrical propagation probabilities from the center pixel. For example, the symmetrical propagation direction from the center pixel can be understood as the directions of the positive and negative X-axis and the positive and negative Y-axis in a two-dimensional coordinate system with the center pixel as the origin. In this way, by using the center pixel as a symmetry point, the cropping probability of pixels in a symmetrical region is determined, ensuring that the cropped region after cropping the training image is located in the middle of the image, thus increasing the probability that the cropped region includes the region of interest.

[0083] In some embodiments, after determining the training clipping region, further optimization of the clipping region can be achieved in the following two ways:

[0084] Method 1: First, in the training image, determine the line that includes the center pixel of the training image.

[0085] Here, after determining the center pixel in the training image, a line is defined including the pixel corresponding to that center pixel. Any line in the training image that passes through the point where the center pixel is located can be used. This line can be vertical, horizontal, or diagonal.

[0086] The second step is to correct the training clipping region into an axisymmetric region with the line as the axis of symmetry.

[0087] Here, the training cropping region is corrected according to this line, so that the cropping region is a line-symmetric region based on the line including the center pixel. In this way, the training cropping region is a line-symmetric region, and the axis of symmetry is this line.

[0088] The first and second steps described above correct the training cropping region to be the axis-symmetric region of the line centered on the pixel in the training image. This makes the corrected training cropping region include the pixels in the central region of the training image, thus making the training cropping region more conducive to gaze estimation.

[0089] Method 2: Correct the training cropping region to a centrally symmetrical region with the central pixel as the center of symmetry.

[0090] Here, the training cropping region is corrected using the center pixel as the symmetry point, so that the corrected training cropping region is a point-symmetric region with the center pixel as the symmetry point. In this way, the corrected training cropping region is more likely to include the central region of the training image, and thus the training cropping region has a greater probability of including the eye image used for gaze estimation.

[0091] In some embodiments, the pruning probability and network weights are optimized by satisfying the value of an objective function that includes the difference between the true value and the output result; that is, step S103 above can be achieved by... Figure 2 The steps shown are to be implemented as follows:

[0092] Step S201: Determine the value of the objective function based on the output result and the true value.

[0093] In some embodiments, the difference between the output and the ground truth of the training image is obtained by comparing the output of the image processing network with the ground truth of the training image; and based on the difference, the loss function of the image processing network for predicting the output is obtained, which is the objective function.

[0094] In some possible implementations, the value of this objective function is the task loss of the image processing network. During training, this task loss, i.e., the value of the objective function, is obtained by comparing the output of the image processing network predicting the entire training image with the output corresponding to the cropped region. This objective function can be a loss function trained on the network parameters of the image processing network itself, such as the weights in the image processing network and the pruning probability of the channel to be pruned. The weights of the image processing network and the transition probabilities of the Markov chain are updated alternately.

[0095] Step S202: Based on the value of the objective function and the computational loss of the image processing network, adjust the network parameter values ​​and the cropping probability to obtain the trained image processing network.

[0096] In some embodiments, the computational loss of the image processing network characterizes the difference between the expected computational cost and the actual computational cost of the image processing network; for example, the difference between the expected and actual computational cost of the image processing network is used as the computational loss. Thus, by combining the value of the objective function and the computational loss of the image processing network, the network parameter values ​​and cropping probabilities of the image processing network are updated alternately to train the image processing network, resulting in a trained image processing network.

[0097] In this embodiment, the value of the objective function is obtained by the difference between the output of the training image and the ground truth. The value of the objective function is combined with the computational loss of the image processing network. The weights or cropping probabilities of the image processing network are optimized by the combined loss, so that the optimized weights and cropping probabilities in the trained image processing network are better.

[0098] In some embodiments, the value of the objective function is obtained by comparing the output of the training cropped region with the output of the entire training image. That is, step S201 above can be achieved by the following steps S211 to S213 (not shown in the figure):

[0099] Step S211: The output result is fused with the ground truth to obtain the first fusion result of the training cropped region.

[0100] In some embodiments, during the training of the image processing network for gaze estimation, the output result corresponding to the training cropping region, i.e., the output result of the image processing network for gaze estimation of the training cropping region, is multiplied element-wise with the ground value of the training image corresponding to the training cropping region to obtain the first fusion result.

[0101] Step S212: The gaze estimation result of the image processing network on the training image is fused with the ground truth to obtain the second fusion result of the training image.

[0102] In some embodiments, during the training of the image processing network for gaze estimation, the output result corresponding to the entire training image, i.e., the output result of the image processing network for gaze estimation of the entire training image, is multiplied element-wise with the ground value of the training image to obtain the second fusion result.

[0103] Step S213: Based on the ratio of the first fusion result to the second fusion result, obtain the value of the objective function.

[0104] In some embodiments, the first fusion result and the second fusion result are serialized respectively, and the ratio between the serialized first fusion result and the second fusion result is determined; thus, for any frame of training image, the expectation of the training image is determined based on the ratio, and the expectation is determined as the value of the objective function.

[0105] In this embodiment of the application, by comparing the first fusion result of the training cropped region and the second fusion result of the training image, the difference between the gaze estimation result of the image processing network for the training cropped region and the gaze estimation result for the entire training image is obtained. This difference is represented as the value of the objective function, thereby enabling the optimization of network parameter values ​​such as weights of the image processing network through the value of the objective function.

[0106] In some embodiments, while optimizing the network parameter values ​​of the image processing network, the channels to be pruned in the image processing network are also optimized. That is, while cropping the pixels of the training image input to the image processing network, the channels of the image processing network are also cropped to obtain a trained image processing network with better performance. That is, the above step S202 can be implemented by the following steps S221 and S222 (not shown in the figure):

[0107] Step S221: Based on the value of the objective function and the computational loss, obtain the transfer loss.

[0108] In some embodiments, during the training of the image processing network, the value of the objective function and the computational loss are summed element-wise to obtain the transfer loss.

[0109] Step S222: Adjust the weights and the pruning probability of the channel to be pruned based on the value of the objective function, and adjust the pruning probability based on the transfer loss to obtain the trained image processing network.

[0110] In some embodiments, the weights of the image processing network are optimized according to the value of the objective function, and the pruning probability of the channels to be pruned is also optimized. This allows for the pruning of channels in the image processing network using the optimized pruning probabilities, thereby compressing the network and making it more lightweight. Simultaneously with channel pruning, the pruning probability of pixels in the input training image is optimized, resulting in pruning of the training image according to the optimized pruning probabilities, thus improving the effectiveness of the training image.

[0111] In this embodiment, the pruning probability is optimized by using a transfer loss to achieve the optimal adjusted pruning probability; the weights and pruning probabilities of the channels to be pruned are optimized by the value of the objective function, thereby performing channel pruning on the image processing network and pixel pruning on the input training image, which improves the accuracy of gaze estimation of the trained image processing network.

[0112] This application provides an image processing method, such as... Figure 3 As shown, combined with Figure 3 The steps shown are explained below:

[0113] Step S301: Obtain the image to be processed.

[0114] Here, the image to be processed can be an image captured in any scene, and can be a simple or complex image. It can be an image acquired in any scene, captured by an image acquisition device such as a camera, or received from other devices. The image to be processed can also be a gaze estimation image, such as a facial image including the eyes, or a human image including the eyes. For example, in a traffic scenario, the gaze estimation image could be a facial image or eye image of the driver, or a facial image or eye image of a passenger in the vehicle. In the application scenario of smart devices, the gaze estimation image could be a facial image or eye image of the person controlling the smart device. By estimating the gaze of this person, the smart device can be controlled based on the direction of that gaze.

[0115] Step S302: Based on the cropping probability of the trained image processing network, perform pixel cropping on the image to be processed to obtain the cropping region to be processed.

[0116] Here, the trained image processing network can be trained based on the image processing network training method provided in the above embodiments. The acquired image to be processed is input into the trained image processing network to obtain the cropping region to be processed. In this image processing network, the optimized cropping probability is multiplied with the vector representing the image to be processed to perform pixel cropping of the image to be processed, and the result of the multiplication is the cropping region to be processed.

[0117] Step S303: The trained image processing network is used to process the cropped region to obtain the processing result of the image to be processed.

[0118] Here, after pixel cropping of the image to be processed, the trained image processing network is used to process the cropped region, which can improve the accuracy of the processing results.

[0119] In some possible implementations, taking the trained image processing network for gaze estimation as an example, after optimizing the cropping probability of the image processing network, the acquired image is cropped using the optimized cropping probability during the gaze estimation process to obtain the gaze estimation result. This can be achieved through the following process:

[0120] The trained image processing network is used to estimate the gaze distance of the cropped region to be processed, thereby obtaining the processing result of the image to be processed.

[0121] Here, in the context of gaze estimation, the optimized cropping probability from the trained image processing network is used to crop the image to be processed, resulting in a cropped region. The image within this cropped region is then the region of interest, such as the eye area. Performing gaze estimation based on this cropped region further improves the accuracy of gaze estimation.

[0122] The following describes the application of the image processing method provided in the embodiments of this application in a real-world scenario, taking the lightweighting of the gaze estimation model as an example.

[0123] In related technologies, channel pruning is used for model acceleration and compression, enabling the deployment of parameterized convolutional neural networks on embedded or mobile devices. In some tasks, computational costs can be reduced by cropping a sub-region of each input image. For example, in appearance-based gaze estimation, each input image is typically normalized so that the center of the eyes or face is located at the center of the image. In this case, it can be assumed that important pixels are distributed around the center of the image, and a sub-region of the input image can be cropped at the center to reduce computational costs. However, in many cases, since it is impossible to determine which region the important pixels are distributed in, a full-face image is used as the gaze estimator. But given the computational constraints, it cannot be determined whether using a full-face image is optimal.

[0124] Based on this, this application introduces a new model acceleration concept, pixel pruning, to find a common optimal region in the input image; and proposes a novel differentiable pixel pruning method called Differentiable Markov Pixel Pruning (DMPP). It searches for redundant pixels in the input image based on task loss and computational constraints. In DMPP, the pixel pruning process is modeled as a Markov chain, and the searched redundant pixels can be pruned during inference through a simple cropping operation; moreover, jointly pruning channels and pixels is more efficient than pruning only channels in gaze estimation.

[0125] In the image processing network training method provided in this application embodiment, multiple Markov chains are used to redefine the pruning process, wherein pixel pruning of multiple Markov chains can be achieved through the following process:

[0126] The input image is represented as I∈[0,1] C×2H×2W Where C, H, and W represent the number of channels, half-height, and half-width of the image, respectively. This application provides four sets of random variables in its embodiments. The state space S corresponding to these random variables +x ,S -x ,S +y ,S -yThis can be expressed as formulas (1) to (4) below:

[0127]

[0128]

[0129]

[0130]

[0131] in, Represents pixel {I i,j |i<W+k},{I i,j |i≥Wk},{I i,j |i<H+k} and {I i,j |i≥Hk} are retained respectively, such as Figure 4 As shown, pixel trimming is modeled as multiple Markov chains. This indicates that k pixels are retained, counting from the center pixel in the +x, -x, +y, and -y directions respectively.

[0132] exist Figure 4 In the diagram, the Markov chain on the X-axis 401 includes the following states: S -x and S +x S -x The states include: S +x The states include:

[0133] The Markov chain on Y-axis 402 includes the following states: S -y and S +y S -y The states include: S +y The states include:

[0134] In this embodiment, the optimal cropping area is marked with a rectangular box, which can be determined by a set of random variables. To represent a cropped rectangle. For example, if the following conditions are met... The cropped rectangle can then be represented as (i min ,j min i max ,j max ), where (i min ,j min ) and (i max ,j max ) represent the coordinates of the bottom left and top right vertices of the rectangle, respectively.

[0135] To learn the optimal region for clipping, the transition probability is parameterized as shown in equations (5) to (8):

[0136]

[0137]

[0138]

[0139]

[0140] Where σ(·) represents a sigmoid function, Let {I} represent the set of learnable parameters. Therefore, the edge probabilities of preserved pixels are {I}. i,j |i<W+k},{I i,j |i≥Wk},{I i,j |i<H+k} and {I i,j |i≥Hk} are respectively represented as As shown in formulas (9) to (12):

[0141]

[0142]

[0143]

[0144]

[0145] In this embodiment, the transition probabilities between states are optimized in an end-to-end manner by multiplying each input image by the edge probability that preserves each pixel. The resulting soft-pruned image is differentiable in terms of transition probabilities, and learnable transition probabilities can be optimized in an end-to-end manner. Given Figure I, the soft-pruned image... This can be expressed as formulas (13) and (14):

[0146]

[0147]

[0148] Here, ⊙ represents the product of elements. For example... Figure 5As shown, the cropping probability of the input image is determined based on Markov chains 51 corresponding to the positive and negative X-axis and Markov chains 52 corresponding to the positive and negative Y-axis. The image is then cropped using this probability to obtain cropped region 501. That is, cropped region 501 represents the cropped image, where each input image is multiplied by the edge probability of retaining each pixel. The generated soft-trimmed image (i.e., cropped region 501) is differentiable in terms of transition probability. The image obtained after soft-trimming pixels (i.e., the cropped region in the above embodiment) is as follows: Figure 6 As shown, multiplying the original image by the learned element-wise edge probabilities of the retained pixels yields... Figure 6 Images 601 to 608 are included, where the target FLOP of images 601 to 604 is 0.5G, and that of images 605 to 608 is also 0.5G. Images 601 and 605 have the same content but different target FLOPs; images 602 and 606 have the same content but different target FLOPs; images 603 and 607 have the same content but different target FLOPs; and images 604 and 608 have the same content but different target FLOPs.

[0149] In this embodiment, the Markov chain is subjected to budget regularization based on differentiable Markov channel pruning, using a set number of FLOPs as the target of budget regularization. For a given image I∈[0,1]... C×2H×2W The desired image size after pixel pruning The calculation is as follows:

[0150]

[0151] Similarly, we can conclude that:

[0152]

[0153] Here, the expected image size is... Calculate the expected FLOP for the entire network. Let FLOP be... s exp(Φ) is the expected FLOP of the entire network. s FLOPtgt represents the target FLOPs, where, The budget regularization loss is shown in Equation (17):

[0154]

[0155] Here, the budget regularization loss can be the computational loss in the above embodiments.

[0156] In this embodiment, the network weights and Markov chain transition probabilities are updated alternately during training. The loss function for the weights is the task loss, specifically the gaze angular loss in this embodiment, as shown in the following formula:

[0157]

[0158] L weight (θ,Φ)=L task (θ,Φ) (19);

[0159] Where I represents the image, g represents the ground truth gaze direction, F(·;θ) represents the gaze estimator parameterized by θ, and P(·;Φ) represents the candidate DMPP cropper parameterized by Φ. Here, the gaze angle loss corresponds to the value of the objective function in the above embodiment. In practice, to update the weights, the gradient is not propagated through P to the transformation parameter Φ because the almost unpruned image sampled from the Markov chain is only used when updating the weights. The loss function for the transition probability Φ (corresponding to the transition loss in the above embodiment) is formulated as follows:

[0160] L trans (θ,Φ)=L task (θ,Φ)+αL budget (Φ) (20);

[0161] In summary, the pixel trimming process is described as multiple Markov chains parameterized by learnable parameters, and can be optimized in an end-to-end manner.

[0162] In this embodiment, redundant pixels in the input image are searched based on task loss and computational constraints. During the differentiable Markov chain pixel pruning process, the pixel pruning process is modeled as multiple Markov chains, and the searched redundant pixels can be pruned during inference through a simple pruning operation.

[0163] In some embodiments, performing channel trimming simultaneously with pixel trimming allows for a better trade-off between utilizing spatial information and network complexity in gaze estimation. For example... Figure 7 As shown, Figure 7 This is a schematic diagram illustrating another application scenario of the image processing network training method provided in the embodiments of this application. Figure 7As can be seen, in the image processing network after channel cropping, channel cropping is performed on the image processing network based on the Markov chain along axis 701 in three-dimensional space, and pixel cropping is performed on the input image based on the Markov chains along axes 702, 703, 704, and 705, resulting in cropping region 706. In this embodiment, by combining the network's channel cropping and the input image's pixel cropping to train the image processing network, the accuracy of image cropping by the image processing network is improved, and the efficiency of gaze estimation is also enhanced.

[0164] Based on the foregoing embodiments, this application provides a training device for an image processing network. The device includes various units and modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0165] Figure 8A This is a schematic diagram of the composition structure of a training device for an image processing network provided in an embodiment of this application, as shown below. Figure 8A As shown, the training device 800 for the image processing network includes:

[0166] The first determining module 801 is used to determine reference pixels based on the training image with labeled ground values;

[0167] The second determining module 802 is used to determine the cropping probability when the image processing network processes the training image, starting from the reference pixel and based on the Markov chain of the training image.

[0168] The first adjustment module 803 is used to adjust the network parameter values ​​and the cropping probability of the image processing network based on the output result of the image processing network processing the training cropping region and the ground truth value, so as to obtain the trained image processing network; the training cropping region is obtained by cropping the training image based on the cropping probability.

[0169] In some embodiments, the training images include: facial images, and the ground truth values ​​are gaze information annotated in the facial images.

[0170] In some embodiments, the labeled line-of-sight information includes at least one of the following: the pitch angle of the line of sight, the yaw angle of the line of sight, and the roll angle of the line of sight.

[0171] In some embodiments, the first determining module 801 includes:

[0172] The first determining submodule is used to determine the center pixel in the training image as the reference pixel;

[0173] The second determining module 802 is further configured to: determine the cropping probability of each pixel in the training image based on the Markov chain, starting from the center pixel.

[0174] In some embodiments, the second determining module 802 includes:

[0175] The second determining submodule is used to determine the transition probability from the center pixel to the next pixel in the Markov chain.

[0176] The third determining submodule is used to determine the cropping probability of the next pixel based on the transition probability of the next pixel and the transition probabilities of multiple pixels preceding the next pixel.

[0177] In some embodiments, the second determining module 802 is further configured to:

[0178] Based on a Markov chain in at least one direction originating from the center pixel, the clipping probability of each pixel in at least one direction originating from the center pixel is set isotropically.

[0179] In some embodiments, the second determining module 802 is further configured to:

[0180] Based on the Markov chain in the symmetrical propagation direction starting from the center pixel, the clipping probability of the pixels in the symmetrical propagation direction starting from the center pixel is set.

[0181] In some embodiments, the apparatus further includes:

[0182] The third determining module is used to determine, in the training image, a line including the center pixel of the training image;

[0183] The first correction module is used to correct the training cropping region into an axisymmetric region with the line as the axis of symmetry after cropping the training image according to the cropping probability to obtain the training cropping region.

[0184] In some embodiments, the apparatus further includes:

[0185] The second correction module is used to correct the training cropping region into a centrally symmetrical region with the central pixel as the center of symmetry after cropping the training image according to the cropping probability to obtain the training cropping region.

[0186] In some embodiments, the first adjustment module 803 includes:

[0187] The fourth determining submodule is used to determine the value of the objective function based on the output result and the truth value;

[0188] The first adjustment submodule is used to adjust the network parameter values ​​and the cropping probability based on the value of the objective function and the computational loss of the image processing network, so as to obtain the trained image processing network.

[0189] In some embodiments, the fifth determining submodule includes:

[0190] The first fusion unit is used to fuse the output result with the ground truth to obtain the first fusion result of the training cropped region;

[0191] The second fusion unit is used to fuse the result of the image processing network processing the training image with the ground truth to obtain the second fusion result of the training image;

[0192] The first comparison unit is used to obtain the value of the objective function based on the ratio of the first fusion result and the second fusion result.

[0193] In some embodiments, the network parameter values ​​of the image processing network include: weights and pruning probabilities of the channels to be pruned, and the first adjustment submodule includes:

[0194] The first determining unit is used to obtain the transfer loss based on the value of the objective function and the computational loss;

[0195] The first adjustment unit is used to adjust the weights and the pruning probability of the channel to be pruned based on the value of the objective function, and to adjust the pruning probability based on the transfer loss, so as to obtain the trained image processing network.

[0196] This application provides an image processing apparatus. Figure 8B This is a schematic diagram of the composition structure of an image processing device provided in an embodiment of this application, as shown below. Figure 8B As shown, the image processing apparatus 820 includes:

[0197] The first acquisition module 821 is used to acquire the image to be processed;

[0198] The first cropping module 822 is used to crop pixels of the image to be processed based on the cropping probability of the trained image processing network to obtain the cropping region to be processed; wherein the trained image processing network is trained based on the training method of the above-mentioned image processing network.

[0199] The first processing module 823 is used to process the cropped region to be processed using the trained image processing network to obtain the processing result of the image to be processed.

[0200] In some embodiments, the trained image processing network is used to perform gaze estimation on the image, and the first processing module 823 is further used to: use the trained image processing network to perform gaze estimation on the cropped region to be processed, and obtain the processing result of the image to be processed.

[0201] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0202] It should be noted that, in the embodiments of this application, if the above-described image processing network training method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0203] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0204] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0205] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0206] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0207] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0208] It should be noted that, Figure 9 This application provides a hardware entity diagram of a computer device as an embodiment of the present application, such as... Figure 9 As shown, the hardware entity of the computer device 900 includes: a processor 901, a communication interface 902, and a memory 903, wherein:

[0209] Processor 901 typically controls the overall operation of computer device 900.

[0210] Communication interface 902 enables computer devices to communicate with other terminals or servers over a network.

[0211] The memory 903 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 901 and various modules in the computer device 900. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 901, the communication interface 902, and the memory 903 can be performed via bus 904.

[0212] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0213] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0214] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0215] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0216] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0217] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0218] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0219] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A training method for an image processing network, characterized in that, The method includes: Based on the labeled ground truth training image, determine the reference pixel; Starting from the reference pixel, the cropping probability of the image processing network when processing the training image is determined based on the Markov chain of the training image, wherein the Markov chain of the training image is a Markov chain that models the cropping process of the training image. Based on the output of the image processing network processing the training cropped region and the ground truth value, the network parameter values ​​of the image processing network and the cropping probability are adjusted to obtain the trained image processing network; the training cropped region is obtained by cropping the training image based on the cropping probability. The training image based on labeled ground truth is used to determine reference pixels, including: The center pixel in the training image is determined as the reference pixel; The step of determining the cropping probability when the image processing network processes the training image based on the Markov chain of the training image, starting from the reference pixel, includes: Starting from the center pixel, the cropping probability of each pixel in the training image is determined based on the Markov chain.

2. The method according to claim 1, characterized in that, The training images include: facial images, and the ground truth values ​​are the gaze information annotated in the facial images.

3. The method according to claim 2, characterized in that, The line-of-sight information indicated includes at least one of the following: The pitch angle of the line of sight, the yaw angle of the line of sight, and the roll angle of the line of sight.

4. The method according to claim 1, characterized in that, The step of determining the cropping probability of each pixel in the training image based on the Markov chain, starting from the center pixel, includes: In the Markov chain, the transition probability from the center pixel to the next pixel is determined; The cropping probability of the next pixel is determined based on the transition probability of the next pixel and the transition probabilities of the multiple pixels preceding the next pixel.

5. The method according to claim 4, characterized in that, The step of determining the cropping probability of each pixel in the training image based on the Markov chain, starting from the center pixel, includes: Based on a Markov chain in at least one direction originating from the center pixel, the clipping probability of each pixel in at least one direction originating from the center pixel is set isotropically.

6. The method according to claim 4, characterized in that, The step of determining the cropping probability of each pixel in the training image based on the Markov chain, starting from the center pixel, includes: Based on the Markov chain in the symmetrical propagation direction starting from the center pixel, the clipping probability of the pixels in the symmetrical propagation direction starting from the center pixel is set.

7. The method according to any one of claims 1 to 6, characterized in that, After cropping the training image according to the cropping probability to obtain the training cropping region, the method further includes: In the training image, a line including the center pixel of the training image is determined; The training clipping region is corrected to an axisymmetric region with the line as the axis of symmetry.

8. The method according to any one of claims 1 to 6, characterized in that, After cropping the training image according to the cropping probability to obtain the training cropping region, the method further includes: The training cropping region is corrected to a centrally symmetrical region with the central pixel as the center of symmetry.

9. The method according to any one of claims 1 to 6, characterized in that, The process of processing the training cropping region based on the image processing network output and the ground truth value, adjusting the network parameter values ​​and the cropping probability of the image processing network to obtain the trained image processing network includes: Based on the output and the true value, determine the value of the objective function; Based on the value of the objective function and the computational loss of the image processing network, the network parameter values ​​and the cropping probability are adjusted to obtain the trained image processing network.

10. The method according to claim 9, characterized in that, Determining the value of the objective function based on the output result and the truth value includes: The output result is fused with the ground truth value to obtain the first fusion result of the training cropped region; The result of the image processing network processing the training image is fused with the ground truth to obtain a second fusion result of the training image; The value of the objective function is obtained based on the ratio of the first fusion result to the second fusion result.

11. The method according to claim 9 or 10, characterized in that, The network parameter values ​​of the image processing network include: weights and pruning probabilities of the channels to be pruned. The network parameter values ​​and pruning probabilities are adjusted based on the value of the objective function and the computational loss of the image processing network to obtain the trained image processing network, including: Based on the value of the objective function and the computational loss, the transfer loss is obtained; The weights and pruning probabilities of the channels to be pruned are adjusted based on the value of the objective function, and the pruning probability is adjusted based on the transfer loss to obtain the trained image processing network.

12. An image processing method, characterized in that, The method includes: Obtain the image to be processed; The image to be processed is cropped pixel by pixel based on the cropping probability of the trained image processing network to obtain the cropping region to be processed; wherein the trained image processing network is trained based on the method described in any one of claims 1 to 11; The trained image processing network is used to process the cropped region to obtain the processing result of the image.

13. The method according to claim 12, characterized in that, The trained image processing network is used to estimate the gaze distance of the image. The process of using the trained image processing network to process the cropped region to obtain the processing result of the image includes: The trained image processing network is used to estimate the gaze distance of the cropped region to be processed, thereby obtaining the processing result of the image to be processed.

14. A training device for an image processing network, characterized in that, include: The first determination module is used to determine reference pixels based on the training image with labeled ground truth values; The second determining module is used to determine the cropping probability when the image processing network processes the training image, starting from the reference pixel and based on the Markov chain of the training image, wherein the Markov chain of the training image is a Markov chain that models the process of cropping the training image. The first adjustment module is used to adjust the network parameter values ​​and the cropping probability of the image processing network based on the output result of processing the training cropping region by the image processing network and the ground truth value, thereby obtaining the trained image processing network; the training cropping region is obtained by cropping the training image based on the cropping probability. The first determining module includes: The first determining submodule is used to determine the center pixel in the training image as the reference pixel; The second determining module is further configured to: determine the cropping probability of each pixel in the training image based on the Markov chain, starting from the center pixel.

15. An image processing apparatus, characterized in that, include: The first acquisition module is used to acquire the image to be processed; The first cropping module is used to crop the image to be processed by pixels based on the cropping probability of the trained image processing network to obtain the cropping region to be processed; wherein the trained image processing network is trained based on the method described in any one of claims 1 to 11; The first processing module is used to process the cropped region to be processed using the trained image processing network to obtain the processing result of the image to be processed.

16. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 11, or when the processor executes the program, it implements the steps of the method according to claim 12 or 13.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 11, or when executed by a processor, the computer program implements the steps of the method according to claim 12 or 13.

Images