Target image generation method and device, computer device, and storage medium

Abstract

The application relates to a target image generation method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: a server first acquires a to-be-processed image, then inputs the to-be-processed image into a pre-trained depth estimation model, encodes the to-be-processed image through the depth estimation model to obtain first color channel features of each pixel point in images of different scales, decodes the first color channel features to obtain second color channel features corresponding to each pixel point in the images of different scales, and finally performs jump connection on the first color channel features and the second color channel features to obtain splicing features corresponding to each pixel point in the images of different scales. In this way, high-level features and low-level features are considered at the same time, the output result is more accurate, that is, a depth value corresponding to each pixel point in the to-be-processed image, and finally a target image corresponding to the to-be-processed image can be obtained according to the depth value.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for generating target images. Background Technology

[0002] With the development of artificial intelligence, target image generation methods have emerged. These methods are crucial for reconstructing 3D scenes and acquiring geometric information from 2D RGB images. In the field of autonomous driving, depth information is essential for planning driving routes and determining driving modes. Currently, depth is primarily obtained using physical devices such as infrared sensors, but these sensors present numerous inconveniences in terms of both cost and coverage. In the era of booming intelligent robots, accurate target image generation methods combined with ordinary RGB images play a decisive role in generating 3D point clouds for robot trajectory control.

[0003] However, existing technologies struggle to accurately predict the depth of small objects, and small objects may even be lost. Summary of the Invention

[0004] Therefore, it is necessary to provide a target image generation method, apparatus, computer device, computer-readable storage medium, and computer program product that can obtain accurate target images to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for generating a target image, the method comprising:

[0006] Obtain the image to be processed;

[0007] The image to be processed is input into a pre-trained depth estimation model. The depth model encodes the image to obtain the first color channel features of each pixel in the image at different scales. The first color channel features are decoded to obtain the second color channel features of each pixel in the image at different scales. The first color channel features and the second color channel features are skip-connected to obtain the stitched features corresponding to each pixel in the image at different scales. The depth value corresponding to each pixel in the image to be processed is calculated based on the stitched features. The target image is obtained based on the depth value.

[0008] In one embodiment, after decoding the first color channel features to obtain the second color channel features of each pixel in an image at different scales, the method further includes:

[0009] The edges of the decoded images at different scales are extended to match the scale of the corresponding images during the encoding process.

[0010] In one embodiment, before inputting the image to be processed into a pre-trained depth estimation model, the method further includes:

[0011] Acquire training data, which carries training labels;

[0012] A deep model is obtained by training the training data;

[0013] Obtain the preset sample set data;

[0014] By adjusting the parameters of the depth model using a preset sample set of data, a depth estimation model is obtained.

[0015] In one embodiment, before obtaining the image to be processed, the method further includes: performing a first preprocessing step on the image to be processed;

[0016] After obtaining the training set data, the process also includes: performing a first preprocessing step on the training set data;

[0017] After obtaining the preset sample data, the process also includes: performing a first preprocessing on the preset sample data;

[0018] The first preprocessing includes at least one of scaling, flipping, and normalization.

[0019] In one embodiment, after obtaining the training set data, the method further includes: performing a second preprocessing on the training labels;

[0020] The training labels undergo a second preprocessing step, which includes converting the training labels into single-channel images and converting the depth values ​​in the depth channel into floating-point numbers.

[0021] Scaling and normalizing a single-channel image, or at least one of these.

[0022] In one embodiment, training the training data to obtain a deep model includes:

[0023] The training data is input into the deep model to be trained to obtain the model output data;

[0024] A target optimization function is generated based on the model output data and sample labels, and the deep model to be trained is optimized based on the target optimization function to obtain the deep model.

[0025] The process of generating the objective optimization function includes:

[0026] Obtain the weight coefficients;

[0027] Calculate the depth loss function between the model output data and the training labels, the gradient loss function between the model output data and the training labels, and the similarity loss function between the model output data and the training labels;

[0028] The target loss function is obtained by calculating the weight coefficients, depth loss function, gradient loss function, and similarity loss function.

[0029] Secondly, this application provides a method for generating a target image of a football city, the method comprising:

[0030] Acquire images of a football field;

[0031] The football field image is processed according to the target image generation method in any one of the embodiments to obtain a target image corresponding to the football field image.

[0032] Thirdly, this application also provides a target image generation apparatus, the apparatus comprising:

[0033] The image acquisition module is used to acquire the image to be processed.

[0034] The target image generation module is used to input the image to be processed into a pre-trained depth estimation model, so that the depth model encodes the image to be processed to obtain the first color channel features of each pixel in the image at different scales, decodes the first color channel features to obtain the second color channel features of each pixel in the image at different scales, performs a skip connection between the first color channel features and the second color channel features to obtain the stitched features corresponding to each pixel in the image at different scales, calculates the depth value corresponding to each pixel in the image to be processed based on the stitched features, and obtains the target image based on the depth value.

[0035] Fourthly, this application also provides a method for generating a football field target image, the apparatus comprising:

[0036] A football field image acquisition module is used to acquire images of football fields.

[0037] The football field target image generation module is used to process the football field image according to the device in any of the above embodiments to obtain a target image corresponding to the football field image.

[0038] Fifthly, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the above embodiments.

[0039] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0040] In a seventh aspect, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0041] The aforementioned target image generation method, apparatus, computer device, storage medium, and computer program product first acquire the image to be processed, then encode the image to be processed while downsampling to obtain a first feature, so as to reduce the dimensionality of the features of the image to be processed and retain effective information, thereby avoiding overfitting. Then, the first feature is decoded while upsampling to enlarge the image. Finally, the obtained second feature is convolved to obtain the target image corresponding to the image to be processed, so as to achieve accurate estimation of small objects and avoid the loss of small objects in the target image generation method. Attached Figure Description

[0042] Figure 1 This is an application environment diagram of the target image generation method in one embodiment;

[0043] Figure 2 This is a flowchart illustrating a target image generation method in one embodiment;

[0044] Figure 3 This is a schematic diagram of a pre-trained depth estimation model network framework in one embodiment;

[0045] Figure 4 This is a schematic diagram of the ResNet network framework in one embodiment.

[0046] Figure 5 This is a flowchart illustrating a method for generating a target image of a football field in one embodiment;

[0047] Figure 6 This is a schematic diagram of a football field target image in one embodiment;

[0048] Figure 7 This is a structural block diagram of a target image generation device in one embodiment;

[0049] Figure 8 This is a structural block diagram of a football field target image generation device in one embodiment;

[0050] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] The target image generation method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 first acquires the image to be processed, and then inputs it into a pre-trained depth estimation model to obtain the target image corresponding to the image to be processed. Specifically, the depth estimation model first encodes the image to be processed to obtain the first color channel features of each pixel in the image at different scales. Then, it decodes the first color channel features to obtain the second color channel features of each pixel in the image at different scales. The first and second color channel features are then connected in a skip connection to obtain the stitched features corresponding to each pixel in the image at different scales. This considers both high-order and low-order features. Finally, the stitched features are calculated to obtain the depth value corresponding to each pixel in the image to be processed. Based on the depth value, the target image can be obtained, thus accurately obtaining the target image. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0053] In one embodiment, such as Figure 2 As shown, a target image generation method is provided, which is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0054] S202, Obtain the image to be processed.

[0055] The image to be processed refers to the image that needs to be processed. It can be any color image, such as a color image taken by a mobile phone or camera. It can be in formats such as .jpg or .gif, and there are no specific restrictions here.

[0056] S204, the image to be processed is input into a pre-trained depth estimation model to encode the image to be processed through the depth model to obtain the first color channel features of each pixel in the image at different scales. The first color channel features are decoded to obtain the second color channel features of each pixel in the image at different scales. The first color channel features and the second color channel features are skip-connected to obtain the stitched features corresponding to each pixel in the image at different scales. The depth value corresponding to each pixel in the image to be processed is calculated based on the stitched features. The target image is obtained based on the depth value.

[0057] The depth estimation model refers to a pre-trained deep learning model that can transform an image to be processed into a target image. The target image is the image output after the image to be processed is processed by the depth estimation model, which can be the depth image corresponding to the image to be processed.

[0058] The first color channel feature refers to the feature after encoding the image to be processed, that is, the feature after feature extraction of the image to be processed. The second color channel feature refers to the feature after decoding the first color channel feature, that is, the feature after restoring the spatial features of the first color channel feature.

[0059] Specifically, the server inputs the image to be processed into a pre-trained depth estimation model, which then obtains the target image corresponding to the image to be processed. The server first encodes the image to be processed to obtain the first color channel features of each pixel in the image at different scales. This is essentially downsampling the image to obtain the extracted first color channel features. While a single downsampling of the image to be processed yields the first color channel features of each pixel in the corresponding scale image, multiple encodings are required to obtain these features for each pixel in the image at different scales. For example, the image to be processed can be upsampled five times to obtain the first color channel features of each pixel in five different sized images.

[0060] Specifically, after obtaining the first color channel features, these features are decoded to obtain the second color channel features for each pixel in the image at different scales. This involves upsampling the first color channel features once, resulting in the magnified second color channel features. Multiple decoding operations are required to obtain the second color channel features for each pixel in the image at different scales. It's important to note that the number of encoding and decoding operations must be consistent. For example, if the image to be processed is encoded 5 times, the corresponding number of decoding operations must also be 5 times. The number of encoding and decoding operations needs to be selected based on the actual application scenario, as excessive encoding and decoding operations can affect the accuracy of the final output.

[0061] Specifically, after obtaining the first and second color channel features, a skip concatenation is performed between them to obtain the stitched features. Skip concatenation refers to combining features from the encoding and decoding layers, thus considering both high-level and low-level features, resulting in a more accurate output: the depth value corresponding to each pixel. Then, the stitched features are calculated, specifically transforming them into a single-channel output (since the image to be processed is a color image, both the first and second color channel features are three-channel features). The output is the pixel's depth value, which, based on the depth value, yields the target image, i.e., the depth value corresponding to the image to be processed. This depth value is obtained through backpropagation using the training labels and the target loss function.

[0062] In one embodiment, combined Figure 3 As shown, Figure 3 This is a schematic diagram of a pre-trained depth estimation model network framework in one embodiment. In this embodiment, the depth estimation model adopts the Res-Unet framework, which includes an encoding module, a decoding module, and a computation module.

[0063] Firstly, the encoding module can use ResNet34 (a residual network) as the backbone to encode the image to be processed. ResNet34 has four large modules: [(3x3,64),(3x3,64)]x3, [(3x3,128),(3x3,128)]x4, [(3x4,256),(3x3,256)]x6, and [(3x3,512),(3x3,512)]x3. Specifically, [the specific combination...] Figure 4 As shown, Figure 4This is a schematic diagram of a ResNet network framework in one embodiment. Layer name represents an intermediate layer, output size represents the output size, max pooling is used for max pooling, and average pooling is used for average pooling. In this embodiment, the stride of one convolutional layer in each large module can be set to 2 to achieve downsampling. Downsampling with a total of 5 convolutional layers can sample the original (640, 480) image to (20, 15), resulting in the first color channel features of the image after 5 layers of downsampling. This significantly reduces memory usage and provides a basis for connecting the first and second color channel features. Furthermore, the 34-layer depth makes feature extraction more efficient. Using ResNet34 as the decoder ensures sufficient network depth for extracting depth features. Due to residual connections, identity mappings can be learned, avoiding gradient explosion and vanishing gradient problems in deep networks. In other embodiments, other decoding methods can be used to decode the image to be processed.

[0064] Secondly, the decoding module can adopt the Unet decoding structure from Res-Unet, namely a 5-layer upsampling module. Each layer first performs bilinear interpolation upsampling (or transposed convolution operation), doubling the size with each upsampling, corresponding to the output of the five downsampling layers. This yields the second color channel features after decoding the first color channel features. In other embodiments, after obtaining the second color channel features, the edges of the decoded images at different scales are expanded to match the scale of the corresponding images during encoding. Then, a skip-connection method is used to connect the encoded and decoded features along the channel dimension. It should be noted that this embodiment not only uses residual connections in the encoding module but also between downsampling and upsampling, employing a multi-scale feature fusion method. The decoding module uses the same architecture as Unet, with corresponding adjustments to the convolution kernel and stride to improve coupling while maintaining the resolution of the depth map.

[0065] Finally, the computation module can use the segmentation head at the end of the Res-Unet network framework. Through a 1x1 convolutional layer, the input (multi-channel) is transformed into a single-channel output. The output is the depth value of the pixel, and the target image corresponding to the image to be processed can be obtained based on the depth value.

[0066] In the above target image generation method, the server first acquires the image to be processed, and then inputs the image to be processed into a pre-trained depth estimation model. The depth estimation model encodes the image to be processed to obtain the first color channel features of each pixel in the image at different scales. The first color channel features are then decoded to obtain the second color channel features corresponding to each pixel in the image at different scales. Finally, the first color channel features and the second color channel features are skip-connected to obtain the stitched features corresponding to each pixel in the image at different scales. This method considers both high-level and low-level features, making the output result more accurate, namely the depth value corresponding to each pixel in the image to be processed. Finally, the target image corresponding to the image to be processed can be obtained based on the depth value.

[0067] In one embodiment, after decoding the first color channel features to obtain the second color channel features of each pixel in the image at different scales, the method further includes: edge extension of the decoded images at different scales to make them the same scale as the corresponding images in the encoding process.

[0068] Specifically, encoding the image to be processed will reduce the image dimension and increase the number of channels. Decoding will increase the image dimension and reduce the number of channels. Therefore, in order to stitch together the first color channel features and the second color channel features, it is necessary to expand the encoding of the images of different scales obtained by decoding so that they are the same as the corresponding image scales in the encoding process. Optionally, zero-padding can be used to pad the edges of the images of different scales obtained by decoding, that is, to fill the image edges with 0.

[0069] In the above embodiments, edge expansion of the decoded images at different scales provides a basis for stitching between the first color channel features and the second color channel features.

[0070] In one embodiment, before inputting the image to be processed into a pre-trained depth estimation model, the method further includes: acquiring training data carrying training labels; training the training data to obtain a depth model; acquiring a preset sample set of data; and adjusting the parameters of the depth model using the preset sample set of data to obtain a depth estimation model.

[0071] Here, training data refers to the data needed to train the deep learning model, which can be an open-source dataset, such as the NYU-Depth V2 dataset. Preset sample set data refers to the samples needed to train the corresponding depth estimation model. For example, if the trained depth estimation model processes images of a football field, then the preset sample set data would be real-world images of a football field. In other embodiments, the specific settings can be tailored to the actual scenario.

[0072] Specifically, the server first obtains training data, which carries training labels, and then inputs the training data with training labels into the initial model. The initial model can be any deep learning model. For example, Res-Unet can be selected as the initial model and trained according to the training labels to obtain a deep model.

[0073] Specifically, after obtaining the deep model, a preset sample set of data is input into the deep model for training. Since the deep model has already learned sufficient decoding and encoding capabilities for depth estimation through training on the training data, the effect of training on the deep model using the preset sample set of data is very significant. When the server trains the deep model on the preset sample set of data, it only needs to adjust the parameters of the deep model to obtain the depth estimation model. In one embodiment, only the parameters of the last layer need to be adjusted, i.e., the parameters of the computation module. It should be noted that the training data and the preset sample set of data need to be similar.

[0074] In one embodiment, training a depth estimation model for processing football field images can involve first training the network on the publicly available NYU-Depth v2 depth estimation dataset, then fine-tuning it on a football dataset, and finally training it again on the football dataset before testing it in real-world football scenes to observe the network's performance. Since the football training set contains only a small amount of sample data, training the entire network solely with this data is insufficient. For a large network like Res-Unet, overfitting will occur, inevitably leading to poor performance in both testing and real-world football scenarios. Therefore, in this embodiment, the entire network is first trained using the NYU-Depth v2 dataset to acquire basic depth feature extraction and recognition capabilities, and then, based on this initialization, trained on the entire football dataset.

[0075] It's important to note that due to the characteristics of game modeling, the NYU-Depth v2 dataset exhibits large abrupt changes in pixel values, resulting in vibrant pixel colors and an unrealistic representation of lighting and occlusion relationships. In contrast, in the real world, lighting plays a crucial role in depth, and depth abrupt changes are relatively smooth and continuous compared to game data. Therefore, this embodiment uses a smaller amount of NYU-Depth v2 data, shuffled, to fine-tune the network, rather than just the football data. The final result is quite excellent considering that real football scenes and the distribution of NYU-Depth v2 data, as well as the football data itself, are almost completely different.

[0076] In the above embodiments, the server first trains the training data to obtain a deep model, and then trains the data on a preset sample set to obtain a depth estimation model. This can solve the problem that the training results are too poor when the preset sample set data is small.

[0077] In one embodiment, before acquiring the image to be processed, the method further includes: performing a first preprocessing on the image to be processed; after acquiring the training set data, the method further includes: performing a first preprocessing on the training set data; after acquiring the preset sample data, the method further includes: performing a first preprocessing on the preset sample data; the first preprocessing includes at least one of scaling, flipping, and normalization.

[0078] The first preprocessing refers to the method of processing the image to be processed and the preset sample data. Specifically, the server first scales and flips the image to be processed and the preset sample data, and then normalizes them. In other embodiments, the first preprocessing can be any one of scaling, flipping, and normalization. It should be noted that the first preprocessing includes at least normalization.

[0079] In one embodiment, for the images to be processed during training, to reduce computation and ensure size invariance, the images are first scaled to a uniform size (640x480). Then, to ensure robustness of the data during prediction and sufficient data for training, the following data augmentation is performed: random horizontal flipping, random rotation [-5°, 5°], and random data augmentation. Finally, the processed images need to be normalized, converting the values ​​in the three channels of the image to floating-point numbers between 0 and 1. In other embodiments, the scaling size and flipping angle of the images can be adjusted according to the actual situation.

[0080] In the above embodiments, performing a first preprocessing on the image to be processed and the preset sample data can ensure better image processing.

[0081] In one embodiment, after acquiring the training set data, the method further includes: performing a second preprocessing on the training labels; the second preprocessing on the training labels includes: converting the training labels into single-channel images and converting the depth values ​​in the depth channel into floating-point numbers; scaling and normalizing the single-channel images at least once.

[0082] The second preprocessing refers to the way the training labels are processed. Specifically, the server first converts the training labels into single-channel images, then converts the depth values ​​in the depth channel into floating-point numbers, and then scales or normalizes the single-channel images.

[0083] In one embodiment, for the depth map used for training (training labels), since the values ​​on the RGB channels are the same, the value on the R channel is taken, converting the original four-channel RGBA into a single-channel grayscale image. To prevent loss of precision, the integer depth value is converted to a floating-point number and simultaneously compared with the RGB values. Figure 1 The sample is scaled to 640×480. Since the grayscale values ​​are between 0 and 255, this embodiment divides each value by 255, thereby normalizing the depth label to a floating-point number between 0 and 1.

[0084] In the above embodiments, by performing a second preprocessing on the training labels, the model can be better guided during training.

[0085] In one embodiment, the above-mentioned training of training data to obtain a deep model includes: inputting training data into the deep model to be trained to obtain model output data; generating a target optimization function based on the model output data and sample labels, and optimizing the deep model to be trained according to the target optimization function to obtain a deep model; wherein, the process of generating the target optimization function includes: obtaining weight coefficients; calculating the depth loss function between the model output data and the training labels, the gradient loss function between the model output data and the training labels, and the similarity loss function between the model output data and the training labels; and calculating the target loss function based on the weight coefficients, the depth loss function, the gradient loss function, and the similarity loss function.

[0086] The target loss function is used to optimize the deep model. The server generates the target loss function based on the training labels and the model output data. The target loss function optimizes the model until training is complete, resulting in a deep model.

[0087] Specifically, training data is fed into the deep model to be trained, i.e., the initial model, to obtain the model output data. Then, the target loss function is calculated based on the training labels and the model output data. Finally, the deep model to be trained is optimized based on the target loss function until the model training is completed.

[0088] Specifically, the server first obtains the weight coefficients, which are actually set according to the actual use scenario and are not specifically limited in this embodiment. Then, it calculates the depth loss function between the model output data and the training labels, the gradient loss function between the model output function and the training labels, and the similarity loss function between the model output and the training labels. Finally, it calculates the target loss function based on the weight coefficients and each loss function.

[0089] In one embodiment, considering the differences between the network output logits (model output data) and the labels, the differences in gradients, and the differences in normals, the target loss function can be expressed as follows:

[0090] ε mm =αε depth +λε grad +με ssim (1)

[0091] In the loss function, α, λ, and μ are weight coefficients, which can be manually set hyperparameters. The loss function has three terms in total. Among them, ε... depth The loss term is the L1 norm of the output logits and the label itself. The L2 norm is not used because it causes problems such as uneven transitions and large differences between distant and near objects in the depth map. The L1 norm transition is smoother, and the calculation method is as shown in formula (2), which is beneficial for the network to learn from the data. At the same time, it is more effective in learning features of depth abrupt changes. The weight of the norm loss is set relatively small in this embodiment because the most critical feature to identify in the depth estimation task is actually the depth abrupt change. Therefore, the weight of the L1 norm loss of a single pixel value is small in this embodiment.

[0092]

[0093] In depth estimation tasks, the distribution of depth in an image is characterized by being mostly smooth, with abrupt changes often occurring at depth edges due to image occlusion or other reasons. Therefore, the gradient of the image, i.e., the points of abrupt change, is a crucial aspect to consider in this embodiment. The gradient difference between the output logits and the label is a loss term that needs to be strictly controlled in this embodiment. Therefore, this embodiment designs a loss term ε. grad That is, formula (3) is used to penalize the gradient loss. In calculating the gradient, this embodiment uses the Sobel operator in the x and y directions to obtain the image edges and gradients. Since the gradient difference is important for the task recognition effect, this embodiment gives a large weight to the gradient loss, but this will bring a problem. Because the upper bound of the L1 loss of the gradient at the pixel level is too large, if the weight is too large, it will lead to an imbalance in the loss. Therefore, this embodiment monitors the value of the gradient and finds that in actual training, the loss brought by the gradient is completely within a reasonable range even with a large weight, and the training effect is relatively good.

[0094]

[0095] Meanwhile, the overall structure of the image also needs to be fully considered in this embodiment. For image structural similarity, this embodiment uses structural similarity, i.e., SSIM, as shown in formula (4). In SSIM, the pixel mean is used as the brightness estimate, the standard deviation as the contrast estimate, and the covariance as the measure of structural similarity. The main reason for choosing SSIM in this embodiment is its ability to estimate covariance and standard deviation, i.e., second-order statistics. Introducing second-order statistics to calculate the loss theoretically provides a good improvement for depth data, which is sensitive to changes. Furthermore, SSIM itself is size invariant, meaning that the expected value of the statistics is consistent between large and small images, which is consistent with the size invariance of the task itself in this embodiment. Experimental results also prove its effectiveness.

[0096]

[0097] Furthermore, considering the size invariance of depth estimation itself—that is, the relative depth should not differ due to changes in image size—this embodiment uses mean-level calculations for all three losses when calculating the loss. Specifically, it uses the mean of the pixel difference L1 norm, the mean of the gradient difference L1 norm, and the weighted sum of ssim, rather than summation-level calculations. This is because large-scale and small-scale images should not differ in depth estimation at the loss level; that is, the depth estimation task itself is size invariant. This embodiment aims to output the same (different-sized) depth map for the same (different-sized) image, which better aligns with the size invariance characteristic of the depth estimation task. In summary, the loss calculation formula is as shown in formula (5).

[0098]

[0099] In the above embodiments, a weighted sum of the absolute value loss of the L1 norm between the depth prediction value and the label on a single pixel, the gradient loss, and the structural invariance loss is considered, which can effectively solve the problem of depth vanishing for small target objects.

[0100] In one embodiment, such as Figure 5 As shown, a method for generating a target image of a football field is provided, the method comprising:

[0101] S502, acquire the image of the football field.

[0102] Among them, football field images refer to images of real-world football fields.

[0103] S504 Processes the football field image according to the target image generation method in any embodiment to obtain a target image corresponding to the football field image.

[0104] Specifically, after the server obtains the football field image, it processes the football field image according to the target image generation method in any of the above embodiments to obtain a target image corresponding to the football field image.

[0105] In one embodiment, a specific combination Figure 6 As shown, the depth map of the scene does not show a soccer ball. This is actually due to the training data, not a problem with the model itself. Figure 6 As can be seen from the depth map example, the depth of the soccer ball is completely absent from the label. Therefore, the network trained on this data has low sensitivity to soccer ball depth. Consequently, the model's output should ideally not include a soccer ball. The reason for not using soccer ball data for separate training is that training with soccer ball data would make the model overly sensitive to circular features, leading to inappropriate fitting of the soccer player's head. To achieve recognition of soccer balls in real-world scenarios, training data with real-world soccer ball depth may be needed later.

[0106] In the above embodiments, the depth estimation problem in football matches is effectively solved by data preprocessing and depth estimation models, which can accurately predict the depth of players on the field.

[0107] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0108] Based on the same inventive concept, this application also provides a target image generation apparatus for implementing the target image generation method described above, and a football field target image generation apparatus corresponding to the football field target image generation method. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations of the one or more target image generation apparatuses and football field target image generation apparatuses provided below can be found in the limitations of the target image generation method and football field target image generation method described above, and will not be repeated here.

[0109] In one embodiment, such as Figure 7 As shown, a target image generation device is provided, including: a to-be-processed image acquisition module 100 and a target image generation module 200, wherein:

[0110] The image acquisition module 100 is used to acquire the image to be processed.

[0111] The target image generation module 200 is used to input the image to be processed into a pre-trained depth estimation model, so as to encode the image to be processed through the depth model to obtain the first color channel features of each pixel in the image at different scales, decode the first color channel features to obtain the second color channel features of each pixel in the image at different scales, perform skip connections between the first color channel features and the second color channel features to obtain the stitched features corresponding to each pixel in the image at different scales, calculate the depth value corresponding to each pixel in the image to be processed based on the stitched features, and obtain the target image based on the depth value.

[0112] In one embodiment, the above-mentioned apparatus further includes:

[0113] The extension module is used to extend the edges of the decoded images at different scales to make them the same scale as the corresponding images during the encoding process.

[0114] In one embodiment, the apparatus further includes:

[0115] The training data acquisition module is used to acquire training data, which carries training labels.

[0116] The training module is used to train the deep model from the training data.

[0117] The sample set acquisition module is used to acquire preset sample set data;

[0118] The parameter adjustment module is used to adjust the parameters of the depth model using a preset sample set of data to obtain a depth estimation model.

[0119] In one embodiment, the above-mentioned apparatus further includes:

[0120] The first preprocessing module is used to perform a first preprocessing on the image to be processed, the training set data, and the preset sample data, wherein the first preprocessing includes at least one of scaling, flipping, and normalization.

[0121] In one embodiment, the above-mentioned apparatus includes:

[0122] The second preprocessing module is used to perform a second preprocessing on the training labels, wherein the second preprocessing includes:

[0123] The transformation unit is used to convert training labels into single-channel images and convert depth values ​​in the depth channel into floating-point numbers.

[0124] A processing unit for scaling and normalizing a single-channel image at least once.

[0125] In one embodiment, the training module includes:

[0126] The input unit is used to input training data into the deep model to be trained to obtain the model output data.

[0127] The optimization unit is used to generate a target optimization function based on the model output data and sample labels, and then optimize the deep model to be trained according to the target optimization function to obtain the deep model.

[0128] The optimization unit includes:

[0129] The weight coefficient acquisition sub-unit is used to obtain the weight coefficients.

[0130] The loss function calculation subunit is used to calculate the depth loss function between the model output data and the training labels, the gradient loss function between the model output data and the training labels, and the similarity loss function between the model output data and the training labels.

[0131] The target loss function calculation subunit is used to calculate the target loss function based on the weight coefficients, depth loss function, gradient loss function, and similarity loss function.

[0132] In one embodiment, such as Figure 8 As shown, a football field target image generation device is provided, comprising:

[0133] A football field image acquisition module is used to acquire images of football fields.

[0134] The football field target image generation module is used by the apparatus in any of the above embodiments to process the football field image and obtain a target image corresponding to the football field image.

[0135] The various modules in the aforementioned target image generation device and the football field target image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0136] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores image data to be processed and images of a football field. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a method for generating a target image and a method for generating a target image of a football field.

[0137] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the above embodiments.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0140] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating a target image, characterized in that, The method includes: Acquire the image to be processed; the image to be processed is a color image; The image to be processed is input into a pre-trained depth estimation model. The depth model encodes the image to obtain the first color channel features of each pixel at different scales. The first color channel features are then decoded to obtain the second color channel features of each pixel at different scales. Skip connections are made between the first and second color channel features to obtain the stitched features corresponding to each pixel at different scales. The depth value corresponding to each pixel in the image to be processed is calculated based on the stitched features. The target image is obtained based on the depth value. The first and second color channel features are three-channel features. The encoding of the image to be processed to obtain the first color channel features of each pixel at different scales is implemented using ResNet34. The decoding of the first color channel features to obtain the second color channel features of each pixel at different scales is implemented using Res-UNet. Skip connections refer to stitching together the features from the encoding and decoding layers. The step of calculating the depth value corresponding to each pixel in the image to be processed based on the stitching features, and obtaining the target image based on the depth values, includes: The semantic segmentation head transforms the multi-channel splicing features into a single-channel output, with the output being the depth value of each pixel. The target image is obtained based on the depth value of each pixel.

2. The method according to claim 1, characterized in that, Before inputting the image to be processed into the pre-trained depth estimation model, the method further includes: Acquire training data, wherein the training data carries training labels; A deep model is obtained by training the training data; Obtain the preset sample set data; The parameters of the depth model are adjusted using the preset sample set data to obtain a depth estimation model.

3. The method according to claim 2, characterized in that, Before acquiring the image to be processed, the method further includes: performing a first preprocessing on the image to be processed; After acquiring the training set data, the method further includes: performing a first preprocessing on the training set data; After obtaining the preset sample data, the method further includes: performing a first preprocessing on the preset sample data; The first preprocessing includes at least one of scaling, flipping, and normalization.

4. The method according to claim 2, characterized in that, After acquiring the training set data, the method further includes: performing a second preprocessing on the training labels; The second preprocessing of the training labels includes: The training labels are converted into single-channel images, and the depth values ​​in the depth channel are converted into floating-point numbers; The single-channel image is scaled and normalized by at least one of the following:

5. The method according to claim 2, characterized in that, The process of training the training data to obtain a deep model includes: The training data is input into the deep model to be trained to obtain the model output data; A target optimization function is generated based on the model output data and sample labels, and the deep model to be trained is optimized based on the target optimization function to obtain a deep model. The process of generating the objective optimization function includes: Obtain the weight coefficients; Calculate the depth loss function between the model output data and the training labels, the gradient loss function between the model output data and the training labels, and the similarity loss function between the model output data and the training labels; The target loss function is obtained by calculating based on the weight coefficients, the depth loss function, the gradient loss function, and the similarity loss function.

6. A method for generating a target image of a football field, characterized in that, The method for generating the football field target image includes: Acquire images of a football field; The target image generation method according to any one of claims 1-5 is used to process the football field image to obtain a target image corresponding to the football field image.

7. A target image generation apparatus, characterized in that, The device includes: The image acquisition module is used to acquire the image to be processed; the image to be processed is a color image. The target image generation module is used to input the image to be processed into a pre-trained depth estimation model, so as to encode the image to be processed by the depth model to obtain the first color channel features of each pixel in the image at different scales, decode the first color channel features to obtain the second color channel features of each pixel in the image at different scales, perform skip concatenation on the first color channel features and the second color channel features to obtain the stitched features corresponding to each pixel in the image at different scales, calculate the depth value corresponding to each pixel in the image to be processed based on the stitched features, and obtain the target image based on the depth value; the first color channel features and the second color channel features are three-channel features; the... The encoding of the image to be processed to obtain the first color channel features of each pixel in the image at different scales is implemented using ResNet34; the decoding of the first color channel features to obtain the second color channel features of each pixel in the image at different scales is implemented using Res-UNet; the skip connection refers to concatenating the features of the encoding and decoding layers; the calculation of the depth value corresponding to each pixel in the image to be processed based on the concatenated features, and the obtaining of the target image based on the depth value, includes: converting the multi-channel concatenated features into a single-channel output through a semantic segmentation head, the output result being the depth value of each pixel; and obtaining the target image based on the depth value of each pixel.

8. A method and apparatus for generating a target image of a football field, characterized in that, The device includes: A football field image acquisition module is used to acquire images of football fields. A football field target image generation module is used to process the football field image according to the apparatus of claim 7 to obtain a target image corresponding to the football field image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5 or 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5 or 6.

Images