Model training method and device, quality evaluation method and device, and electronic device

Abstract

The application discloses a model training method, a quality evaluation method, a device and an electronic device. The model training method comprises: obtaining a sample optical image and a sample depth image, the sample optical image containing a target region corresponding to a target object; performing region segmentation on the target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image; calculating the similarity between the actual region and the target region to generate a quality label corresponding to the sample depth image based on the calculated similarity; and training a to-be-trained quality evaluation model based on the sample depth image and the quality label corresponding to the sample depth image to obtain a trained quality evaluation model. The above scheme can obtain a more objective and accurate quality label according to the relationship between the actual region and the target region, improve the efficiency of label addition, and then improve the training effect of model training by improving the quality of training samples of model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a model training method and device, a quality evaluation method and device, and an electronic device. BACKGROUND

[0002] A perception system needs to perceive or recognize the surrounding environment and then make a judgment, which is mainly achieved by acquiring image data through sensing devices such as laser radar, camera, and depth camera. In the current perception system, in addition to using a camera to collect an RGB (Red, Green, Blue) image in a scene, a depth camera is also used to collect a depth image of a target object in the scene, and an RGB-D (RGB-Depth Map) image is used for obstacle detection, salient object detection, and environment perception.

[0003] However, due to the characteristics of the depth sensor and the characteristics of the material and distance of the target object in the scene, the collected depth image often has uneven quality and does not match the target object in the RGB image, and a low-quality depth image usually produces a lot of noise when fused with the RGB data, affecting the decision and judgment of the perception system. Therefore, how to accurately evaluate the quality of the depth image is particularly important. SUMMARY

[0004] The present application provides at least a model training method, a quality evaluation method, a model training device, a quality evaluation device, an electronic device, and a computer readable storage medium.

[0005] The first aspect of the present application provides a model training method, comprising: obtaining a sample optical image and a sample depth image associated with the sample optical image, the sample optical image containing a target region corresponding to a target object; performing region segmentation on the target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image; calculating the similarity between the actual region and the target region to generate a quality label corresponding to the sample depth image based on the calculated similarity; and training a to-be-trained quality evaluation model based on the sample depth image and the quality label corresponding to the sample depth image to obtain a trained quality evaluation model.

[0006] In an embodiment, performing region segmentation on the target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image comprises: obtaining distribution information of depth values contained in the sample depth image; calculating a target segmentation threshold corresponding to the target object contained in the sample depth image based on the distribution information of the depth values; and performing region segmentation on the target object contained in the sample depth image based on the target segmentation threshold to obtain the actual region.

[0007] In an embodiment, the target segmentation threshold corresponding to the target object contained in the sample depth image is calculated based on the distribution information of the depth values, including: calculating the probability corresponding to each depth value based on the distribution information of the depth values; constructing an entropy value function corresponding to the sample depth image based on the probability corresponding to each depth value; and calculating the target segmentation threshold based on the entropy value function.

[0008] In an embodiment, the similarity between the actual region and the target region is calculated to generate a quality label corresponding to the sample depth image based on the calculated similarity, including: performing region mapping processing between the actual region and the target region, calculating the similarity between the actual region and the target region based on the mapping result; and generating the quality label corresponding to the sample depth image based on the calculated similarity.

[0009] In an embodiment, the region mapping processing between the actual region and the target region is performed, and the similarity between the actual region and the target region is calculated based on the mapping result, including: respectively mapping the actual region and the target region to a reference image; obtaining the overlapping area of the actual region and the target region in the reference image, and the area of the target region; and calculating the similarity between the actual region and the target region based on the overlapping area and the area of the target region.

[0010] In an embodiment, the quality label corresponding to the sample depth image is generated based on the calculated similarity, including: obtaining the merged area of the target region and the actual region in the reference image; calculating a similarity constraint parameter based on the overlapping area and the merged area; performing weighted calculation on the similarity and the similarity constraint parameter based on the respective weights of the similarity and the similarity constraint parameter, to obtain an association score between the target region and the actual region; and taking the association score as the quality label corresponding to the sample depth image.

[0011] The second aspect of the present application further provides a quality evaluation method, including: obtaining a depth image to be evaluated; inputting the depth image to be evaluated into a pre-trained quality evaluation model to obtain a quality evaluation result output by the quality evaluation model; wherein the quality evaluation model is obtained based on the above-mentioned model training method.

[0012] The third aspect of the present application provides a model training apparatus, comprising: a sample acquisition module, configured to acquire a sample optical image and a sample depth image associated with the sample optical image, the sample optical image containing a target region corresponding to a target object; a region segmentation module, configured to perform region segmentation on the target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image; a label determination module, configured to calculate a similarity between the actual region and the target region, and generate a quality label corresponding to the sample depth image based on the calculated similarity; and a model training module, configured to train a quality evaluation model to be trained based on the sample depth image and the quality label corresponding to the sample depth image, and obtain a trained quality evaluation model.

[0013] The fourth aspect of the present application further provides a quality evaluation apparatus, comprising: an image acquisition module, configured to acquire a depth image to be evaluated; and a quality evaluation module, configured to input the depth image to be evaluated into a pre-trained quality evaluation model to obtain a quality evaluation result output by the quality evaluation model; wherein the quality evaluation model is obtained based on the above-mentioned model training method.

[0014] The fifth aspect of the present application provides an electronic device, comprising a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the above-mentioned model training method or quality evaluation method.

[0015] The sixth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, the program instructions being executed by a processor to implement the above-mentioned model training method or quality evaluation method.

[0016] The above-mentioned scheme obtains a quality label corresponding to a sample depth image by analyzing the relationship between an actual region and a target region. Compared with manually adding a quality label of a sample depth image, the quality label of the sample depth image generated according to the relationship between the actual region and the target region is more objective and accurate, and the efficiency of label addition is higher, thereby improving the training sample quality of model training and improving the effect of model training.

[0017] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, rather than limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the technical solutions of the present application.

[0019] Figure 1 is an implementation environment schematic diagram of an exemplary embodiment of the model training method of the present application;

[0020] Figure 2 is a flowchart of an exemplary embodiment of the model training method of the present application;

[0021] Figure 3 is a schematic diagram of obtaining a target region corresponding to a sample optical image according to an exemplary embodiment of the present application;

[0022] Figure 4 is a schematic diagram of model training according to an exemplary embodiment of the present application;

[0023] Figure 5 is a schematic diagram of the model structure of the quality evaluation model according to an exemplary embodiment of the present application;

[0024] Figure 6 is a schematic diagram of calculating the similarity according to an exemplary embodiment of the present application;

[0025] Figure 7 is a schematic diagram of model training according to another exemplary embodiment of the present application;

[0026] Figure 8 is a flowchart of an exemplary embodiment of the quality evaluation method of the present application;

[0027] Figure 9 is a block diagram of a model training device according to an exemplary embodiment of the present application;

[0028] Figure 10 is a block diagram of a quality evaluation device according to an exemplary embodiment of the present application;

[0029] Figure 11 is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present application;

[0030] Figure 12 is a structural schematic diagram of a computer readable storage medium according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0032] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to thoroughly understand the present application, but are not intended to limit the present application.

[0033] The term "and / or", used in the present document, only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present document generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" in the present document means two or more than two. In addition, the term "at least one" in the present document means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0034] The model training method provided by the embodiments of the present application will be described below.

[0035] Please refer to Figure 1 , Figure 1 is a schematic diagram of an implementation environment of the model training method in the present application. As Figure 1 indicated, the implementation environment includes a terminal 110 and a server 120, and the terminal 110 and the server 120 can be directly or indirectly connected through wired or wireless communication.

[0036] The terminal 110 can be a mobile phone, a computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc., but is not limited thereto. The terminal 110 can generally refer to one of multiple terminals, and the embodiments of the present application are only exemplified by the terminal 110. Those skilled in the art can know that the number of the above-mentioned terminals can be more or less. For example, the above-mentioned terminals can be only one, or the above-mentioned terminals can be dozens or hundreds, or more, at this time, the implementation environment of the above-mentioned image processing method also includes other terminals. The number and type of the terminal are not limited in the embodiments of the present application.

[0037] The server 120 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0038] Optionally, the server 120 can undertake the main model training work, and the terminal 110 can undertake the secondary model training work; or the server 120 can undertake the secondary model training work, and the terminal 110 can undertake the main model training work; or the server 120 or the terminal 110 can undertake the model training work separately.

[0039] Exemplarily, the terminal 110 sends a model training instruction to the server 120, the model training instruction carrying a sample optical image and a sample depth image associated with the sample optical image, the server 120 performs region segmentation on a target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image, calculates a similarity between the actual region and a target region, generates a quality label corresponding to the sample depth image based on the calculated similarity, trains a quality evaluation model to be trained based on the sample depth image and the quality label corresponding to the sample depth image, obtains a trained quality evaluation model, and then sends the quality evaluation model to the terminal 110 to enable the terminal 110 to deploy the quality evaluation model.

[0040] It can be understood that in the specific embodiments of the present application, data related to sample optical images, sample depth images, user information, etc. are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions.

[0041] Please refer to Figure 2 , Figure 2 is a flowchart of a model training method according to an exemplary embodiment of the present application. The model training method can be applied to the implementation environment shown in Figure 1 and specifically executed by a server in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and specifically executed by devices in other implementation environments, and the present embodiment does not limit the implementation environment to which the method is applied.

[0042] The model training method according to the embodiments of the present application will be described in detail below with the server as a specific execution subject.

[0043] As shown in Figure 2 , in an exemplary embodiment, the model training method at least includes steps S210 to S240, which are described in detail as follows:

[0044] Step S210: Obtain a sample optical image and a sample depth image associated with the sample optical image, the sample optical image containing a target region corresponding to a target object.

[0045] An optical image refers to an image with continuous changes in gray scale and color. Generally, an optical image is an image acquired by an optical photography system, for example, a visible light black-and-white panchromatic image, an RGB color image, a multi-band photography image, a thermal infrared photography image, and the like all belong to optical images. Among them, RGB color is optical trichromatic color, R represents Red, G represents Green, and B represents Blue. Any color visible to the naked eye in nature can be formed by mixing and superimposing the three colors.

[0046] A depth image is also referred to as a range image, which refers to an image taking the distance (depth) from a depth sensor to each point in a scene as a pixel value, and directly reflects the geometric shape of the visible surface of the scene.

[0047] The sample optical image is associated with at least one sample depth image, and the sample depth image associated with the sample optical image refers to an image containing at least the same target object as the sample optical image. For example, the image acquisition device is simultaneously provided with an optical camera and a depth camera, the optical camera is used to acquire an RGB image corresponding to a scene, and the depth camera is used to acquire a depth image corresponding to the scene. At t1, the optical camera acquires a sample optical image a1, and at t1, the depth camera acquires a sample depth image b1, so the sample depth image b1 can be used as an image associated with the sample optical image a1. It should be noted that the above judgment of whether the sample optical image and the sample depth image are associated is only an example, and the judgment mode can be flexibly set according to the actual application scene. For example, in addition to judging whether the sample optical image and the sample depth image are associated from the acquisition time of the image, the posture (such as the pitch angle, the shooting direction, etc.) of the optical camera and the depth camera for acquiring the image can also be used to judge whether the sample optical image and the sample depth image are associated, which is not limited in the present application.

[0048] The sample optical image contains a target region corresponding to a target object. The target object can be a salient target object in the sample optical image, and the salient target refers to the most eye-catching object in the image. The target object can also be a specified object in the sample optical image, such as an obstacle, a moving object, and the like, and the specific type of the target object is not limited in the present application.

[0049] In some embodiments, the target object contained in the sample optical image can be manually labeled by a person, or the target object contained in the sample optical image can be labeled by a computer using a preset region division algorithm.

[0050] For example, refer to Figure 3 , Figure 3As shown in the schematic diagram of obtaining a target region corresponding to a target object contained in a sample optical image shown in an exemplary embodiment of the present application, taking a target object as a salient target as an example, the sample optical image is input into a pre-trained salient target detection model to obtain a detection result output by the salient target detection model. The detection result is used to represent an image region of the salient target contained in the sample optical image, and the image region of the salient target is marked as the target region. Figure 3

[0051] After receiving the instruction of model training, the server obtains a sample optical image corresponding to the quality evaluation model to be trained and a sample depth image associated with the sample optical image.

[0052] For example, the storage device corresponding to the server can store a plurality of sample optical images and sample depth images associated with each sample optical image. The sample optical image and the sample depth image associated with the sample optical image are obtained by querying the storage device. Alternatively, after receiving the request of model training, the server can send a request for obtaining the sample optical image and the sample depth image to an image acquisition device. The image acquisition device acquires the sample optical image and the sample depth image in response to the request. The server obtains the sample optical image acquired by the image acquisition device and the sample depth image corresponding to each sample optical image. The present application does not limit the manner of obtaining the sample optical image and the sample depth image.

[0053] Step S220: performing region segmentation on the target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image.

[0054] Region segmentation refers to classifying pixels in an image to divide the image into at least two different regions.

[0055] The target object contained in the sample depth image is subjected to region segmentation, and the region corresponding to the target object contained in the sample depth image is taken as the actual region. It should be emphasized that the target object corresponding to the target region of the sample optical image and the target object contained in the sample depth image refer to the same object.

[0056] For example, the target object contained in the sample depth image can be subjected to region segmentation based on a pre-set region segmentation algorithm, or subjected to region segmentation based on a pre-trained depth image region segmentation model. The present application does not limit this.

[0057] ​In some embodiments, before the target object contained in the sample depth image is regionally segmented, the sample depth image can also be pre-processed to regionally segment the target object contained in the pre-processed sample depth image. The purpose of pre-processing is to enhance the useful information in the image and remove the useless information in the image to improve the subsequent image processing effect. The pre-processing methods include but are not limited to depth image enhancement, denoising, cropping, etc. For example, the pre-processing of the sample depth image can include: converting the sample depth image into point cloud data, using a ground segmentation algorithm (Random Sample Consensu, RANSAC) to segment the ground area and the non-ground area in the sample depth image, to remove the ground area in the sample depth image, retain the target object and the background in the sample depth image, and improve the accuracy of subsequent regional segmentation of the target object contained in the sample depth image.

[0058] Step S230: calculating the similarity between the actual region and the target region to generate the quality label corresponding to the sample depth image based on the calculated similarity.

[0059] The similarity between the actual region and the target region can refer to the similarity between the actual region and the target region in terms of region contour, region area, and position of the region in its corresponding image, etc. Therefore, the similarity between the actual region and the target region can be calculated based on at least one of the region contour, the region area, and the position of the region in its corresponding image of the actual region and the target region.

[0060] It can be understood that the higher the similarity between the actual region and the target region, the higher the quality of the sample depth image, and vice versa, the lower the similarity between the actual region and the target region, the lower the quality of the sample depth image.

[0061] By calculating the similarity between the actual region and the target region to generate the quality label corresponding to the sample depth image based on the calculated similarity, the quality label of the sample depth image generated by the present application according to the relationship between the actual region and the target region is more objective and accurate than manually adding the quality label of the sample depth image, and the efficiency of label addition is higher, thereby improving the training sample quality of model training to improve the effect of model training.

[0062] Step S240: training the quality evaluation model to be trained based on the sample depth image and the quality label corresponding to the sample depth image to obtain the trained quality evaluation model.

[0063] The quality evaluation model to be trained can be a model that has been iteratively trained for a certain number of times, or an initialized model, which is not limited by the present application.

[0064] The trained quality evaluation model refers to a model that can perform quality evaluation on an input depth image.

[0065] Referring to Figure 4 , Figure 4 A schematic diagram of model training is shown for an exemplary embodiment of the present application. As shown in Figure 4 , a target region corresponding to a target object in a sample optical image is obtained, and region segmentation is performed on the target object contained in a sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image. A similarity between the actual region and the target region is obtained by comparing the target region and the actual region. A quality label corresponding to the sample depth image is generated based on the similarity. The sample depth image and the quality label corresponding to the sample depth image are used as training samples. Then, the quality label of each sample depth image is labeled by the above-mentioned method to obtain a training sample set. The quality evaluation model to be trained is iteratively trained based on the training sample set to obtain the trained quality evaluation model.

[0066] Referring to Figure 5 , Figure 5 A model structure diagram of the quality evaluation model is shown for an exemplary embodiment of the present application. As shown in Figure 5 , the quality evaluation model includes a feature extraction layer, a fully connected layer, a linear layer, and an output layer. The feature extraction layer can be implemented based on MobileNetV2. MobileNetV2 has fewer network parameters and lower computational cost than a general feature extraction network. It is composed of a convolution module, a reverse residual structure using deep separable convolution, and an average pooling module. After feature extraction of the sample depth image by the feature extraction layer, a feature vector is obtained. Then, the feature vector is subjected to a Flatten operation. Flatten is used to "flatten" the input, i.e., to one-dimensionalize the multi-dimensional feature vector, to transition from the convolution layer to the fully connected layer. The fully connected layer is connected to all nodes of the previous layer, and is used to integrate the features extracted in the front. Further, the features output by the fully connected layer are activated by a PReLu activation function, and a Dropout operation can be used. Dropout prevents overfitting of model training by randomly deleting neural network nodes with a probability p. When the last fully connected layer (linear layer) outputs, the output is activated by a Sigmoid activation function, and the output 0,1] quality score.

[0067] During the model training process, a loss value is calculated based on the quality score of the model output and the quality label corresponding to the sample depth image, and the quality evaluation model to be trained is trained based on the calculated loss value until a model training completion condition is met. The model training completion condition can be that the number of model iterations reaches a preset number, or that the loss value is less than a preset loss value, which is not limited in the present application.

[0068] In some embodiments, the target object contained in the sample depth image is regionally segmented to obtain an actual region corresponding to the target object contained in the sample depth image, including: obtaining distribution information of depth values contained in the sample depth image; based on the distribution information of the depth values, calculating a target segmentation threshold corresponding to the target object contained in the sample depth image; based on the target segmentation threshold, regionally segmenting the target object contained in the sample depth image to obtain the actual region.

[0069] The depth value contained in the sample depth image refers to the value of a pixel point in the sample depth image. In a depth image, the value of each pixel point represents the distance of the object at the specific (x, y) coordinate from the camera plane in the field of view of the depth sensor.

[0070] The target segmentation threshold corresponding to the target object contained in the sample depth image is calculated based on the distribution information of the depth values, so as to regionally segment the target object contained in the sample depth image based on the target segmentation threshold to obtain the actual region.

[0071] Illustratively, the target segmentation threshold corresponding to the sample depth image is calculated based on the distribution information of the depth values, including: based on the distribution information of the depth values, the probability corresponding to each depth value is counted; based on the probability corresponding to each depth value, an entropy value function corresponding to the sample depth image is constructed; and based on the entropy value function, the target segmentation threshold is calculated.

[0072] For example, for a depth image with a depth value range of , the entropy value of its histogram is defined as , wherein represents the probability corresponding to the value of the depth value in the sample depth image.

[0073] Then, the entropy value function corresponding to the sample depth image is constructed according to the probability corresponding to each depth value. The specific construction process can be as follows:

[0074] Suppose that the target segmentation threshold t divides the sample depth image into an actual region O and a background region B, then the probability distribution of the actual region O in the sample depth image is , , and the probability distribution of the background region B in the sample depth image is , , wherein ;

[0075] Based on the probability distribution of the actual region O and the background region B, the entropy function of the actual region O is derived and the entropy function of the background region B is respectively:

[0076]

[0077] Therefore, the entropy value function corresponding to the sample depth image is obtained as:

[0078]

[0079] Based on the entropy value function corresponding to the sample depth image, the optimal threshold T is calculated as:

[0080]

[0081] The optimal threshold T is taken as the target segmentation threshold corresponding to the target object contained in the sample depth image.

[0082] Then, the sample depth image can be binarized according to the target segmentation threshold, for example, the depth values greater than the target segmentation threshold are converted to 0 as the background region, and the depth values less than the target segmentation threshold are converted to 1 as the actual region of the target object contained in the depth image.

[0083] By analyzing the distribution information of the depth values contained in the sample depth image, the target segmentation threshold is calculated, and the sample depth image is binarized according to the target segmentation threshold to segment the target object in the sample depth image, simplifying the segmentation process of the actual region and improving the efficiency and accuracy of the region segmentation.

[0084] It can be understood that the above-mentioned method of calculating the target segmentation value is only illustrative, and other calculation methods can also be used to calculate the target segmentation threshold corresponding to the target object, which is not limited by the present application.

[0085] In some embodiments, the similarity between the actual region and the target region is calculated to generate a quality label corresponding to the sample depth image based on the calculated similarity, including: performing region mapping processing between the actual region and the target region, calculating the similarity between the actual region and the target region based on the mapping result; generating a quality label corresponding to the sample depth image based on the calculated similarity.

[0086] Region mapping processing refers to mapping each pixel in a region to the corresponding position of another image according to certain rules to form a new image.

[0087] The region mapping between the actual region and the target region can be mapping each pixel in the actual region to a corresponding image of the target region, or mapping each pixel in the target region to a corresponding image of the actual region, or constructing a reference image and mapping the actual region and the target region into the reference image respectively.

[0088] After the region mapping of the actual region and the target region, the actual region and the target region are in the same image, so as to facilitate the comparison of the actual region and the target region.

[0089] Exemplarily, the calculation process of the similarity is described by taking the construction of a reference image and the mapping of the actual region and the target region into the reference image as an example. The region mapping between the actual region and the target region is performed, and the similarity between the actual region and the target region is calculated based on the mapping result, including: mapping the actual region and the target region into the reference image respectively; obtaining an overlapping area of the actual region and the target region in the reference image, and a region area of the target region; and calculating the similarity between the actual region and the target region based on the overlapping area and the region area of the target region.

[0090] Referring to Figure 6 , Figure 6 is a schematic diagram of calculating the similarity shown in an exemplary embodiment of the present application. As Figure 6 shown, the actual region corresponding to the target object contained in the sample depth image is mapped into the reference image, and the target region corresponding to the target object contained in the sample optical image is mapped into the reference image, then the overlapping area of the actual region and the target region in the reference image and the region area of the target region are obtained, and the similarity between the actual region and the target region is calculated according to the overlapping area and the region area of the target region.

[0091] The calculation formula of the similarity can be as follows:

[0092]

[0093] wherein, the actual region is represented by A, the target region is represented by B, and the similarity between the actual region and the target region is reflected by the ratio of the area of the intersection of the actual region and the target region to the target region. .

[0094] It should be noted that the actual region and the target region can also be compared without region mapping processing, and the comparison manner of the actual region and the target region can be flexibly selected according to actual application conditions. For example, the sample depth image containing the actual region and the sample optical image containing the target region can be image size aligned, such as adjusting the size of the sample depth image and the sample optical image to be consistent through reduction or enlargement, and then the actual region and the target region in the adjusted sample depth image and sample optical image are compared, such as comparing the area between the actual region and the target region, the position information of the actual region and the target region in the corresponding image, the contour information of the actual region and the target region, and the like, to obtain the similarity between the actual region and the target region.

[0095] Exemplarily, the quality label corresponding to the sample depth image is generated based on the calculated similarity, including: obtaining a merged region area of the target region and the actual region in the reference image; calculating a similarity constraint parameter based on the overlap region area and the merged region area; performing weighted calculation on the similarity and the similarity constraint parameter based on the weights corresponding to the similarity and the similarity constraint parameter respectively, to obtain an association score between the target region and the actual region; and taking the association score as the quality label corresponding to the sample depth image.

[0096] The specific calculation formula can be as follows:

[0097]

[0098] wherein, the actual region is represented by A, the target region is represented by B, the similarity constraint parameter is represented by S.

[0099] The formula for calculating the association score in combination with the similarity and the similarity constraint parameter can be as follows:

[0100]

[0101] wherein, the quality label corresponding to the sample depth image is represented by Q, the weight coefficient corresponding to the similarity is represented by a, and the weight coefficient corresponding to the similarity constraint parameter is 1, because noise and inaccuracy may be caused when the sample depth image is regionally segmented, and the quality label is obtained by combining the similarity and the similarity constraint parameter, so that the integrity of the coverage region is more emphasized. and The quality label is obtained by combining the similarity and the similarity constraint parameter, so that the integrity of the coverage region is more emphasized. For example, a can be set to 0.3 to emphasize that the similarity constraint parameter is more important than the similarity.

[0102] Then, the quality label of each sample depth image is calculated​​ , the normalized quality label score is calculated :

[0103]

[0104] wherein, , is the number of sample depth images.

[0105] In some embodiments, referring to Figure 7 , Figure 7 is a model training schematic diagram shown in an exemplary embodiment of the present application. As shown in Figure 7 , the sample label acquisition process is: using the RANSAC ground segmentation algorithm to perform ground segmentation on the sample depth image, to obtain a non-ground depth map , then the non-ground depth map corresponding to the target segmentation threshold is calculated, to perform binary processing on the non-ground depth map based on the target segmentation threshold, to obtain the actual region corresponding to the target object contained in the sample depth image, and then the correlation score between the target region and the actual region is calculated, and the correlation score is taken as the quality label corresponding to the sample depth image. The model training process is: calculating the loss value based on the quality score output by the model and the quality label corresponding to the sample depth image, and training the quality evaluation model to be trained based on the calculated loss value until the model training completion condition is met.

[0106] wherein, the model training completion condition can be that the number of model iterations reaches a preset number, or the loss value is less than a preset loss value, which is not limited by the present application.

[0107] The loss value can be calculated based on a loss function, and the loss function includes but is not limited to a logarithmic loss function, a cross-entropy loss function, a robust loss function (Huber Loss), etc.

[0108] Taking the robust loss function as an example, the calculation formula of the loss value can be as follows:

[0109]

[0110] wherein, represents the quality label corresponding to the sample depth image input into the model, represents the quality estimation of the regression network and the pseudo-label quality score of the depth map, represents the position parameter, which is taken as 1 during training.

[0111] ​The model training method provided in this application obtains the quality labels corresponding to the sample depth images by analyzing the relationship between the actual region and the target region. Compared with manually adding quality labels to the sample depth images, the quality labels of the sample depth images generated by this application based on the relationship between the actual region and the target region are more objective and accurate, and the label addition efficiency is higher. Thus, the model training effect is improved by improving the quality of the training samples for model training.

[0112] The quality assessment method provided in the embodiments of this application will be described below.

[0113] Please see Figure 8 , Figure 8 This is a flowchart illustrating a quality assessment method in an exemplary embodiment of this application. This model training method can be applied to… Figure 1 The method is implemented in the illustrated environment and is specifically executed by at least one of the servers or terminals in that environment. It should be understood that the method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0114] The following describes in detail the model training method proposed in this application embodiment, using a terminal as the specific execution subject. The terminal is equipped with a quality assessment model trained by the above-described model training method.

[0115] like Figure 8 As shown, in an exemplary embodiment, the quality assessment method includes at least steps S810 to S820, which are described in detail below:

[0116] Step S810: Obtain the depth image to be evaluated.

[0117] For example, a terminal can communicate with an image acquisition device to acquire depth images collected by the image acquisition device, and use the acquired depth images as depth images to be evaluated.

[0118] For example, the terminal stores depth images, and the current depth image to be evaluated can be selected from these depth images.

[0119] Step S820: Input the depth image to be evaluated into the pre-trained quality assessment model to obtain the quality assessment result output by the quality assessment model; wherein, the quality assessment model is obtained based on the model training method described above.

[0120] The terminal deploys the quality evaluation model trained by the model training method, inputs the depth image to be evaluated into the pre-trained quality evaluation model, and obtains a quality evaluation result output by the quality evaluation model. The quality evaluation model can perform quality evaluation on the input depth image, and the process can refer to the related description of the quality evaluation model training process, which is not repeated here.

[0121] It can be understood that the pre-trained quality evaluation model can also be deployed in a server. The terminal can send the depth image to be evaluated to the server, so that the server inputs the depth image to be evaluated into the pre-trained quality evaluation model, obtains a quality evaluation result output by the quality evaluation model, and returns the quality evaluation result to the terminal.

[0122] Through the pre-trained quality evaluation model, the depth image can be more accurately evaluated, and the depth image can be screened based on the quality evaluation result. The high-quality depth image is fused with the optical image to reduce the noise generated when the depth image is fused with the optical image, and the accuracy of subsequent image processing is improved.

[0123] Figure 9 is a block diagram of a model training device according to an example embodiment of the present application. As shown in Figure 9 The example model training device 900 includes a sample acquisition module 910, a region segmentation module 920, a label determination module 930, and a model training module 940. Specifically:

[0124] The sample acquisition module 910 is configured to acquire a sample optical image and a sample depth image associated with the sample optical image, the sample optical image containing a target region corresponding to a target object;

[0125] The region segmentation module 920 is configured to perform region segmentation on the target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image;

[0126] The label determination module 930 is configured to calculate the similarity between the actual region and the target region, and generate a quality label corresponding to the sample depth image based on the calculated similarity;

[0127] The model training module 940 is configured to train a quality evaluation model to be trained based on the sample depth image and the quality label corresponding to the sample depth image, and obtain a trained quality evaluation model.

[0128] In the example model training apparatus, the quality label corresponding to the sample depth image is obtained by analyzing the relationship between the actual region and the target region. Compared with manually adding the quality label of the sample depth image, the quality label of the sample depth image generated according to the relationship between the actual region and the target region is more objective and accurate, and the efficiency of label addition is higher, thereby improving the training sample quality of model training and improving the effect of model training.

[0129] Figure 10 is a block diagram of a quality evaluation apparatus according to an example embodiment of the present application. As shown in Figure 10 the example model training apparatus 1000 includes an image acquisition module 1010 and a quality evaluation module 1020. Specifically:

[0130] The image acquisition module 1010 is configured to acquire a depth image to be evaluated.

[0131] The quality evaluation module 1020 is configured to input the depth image to be evaluated into a pre-trained quality evaluation model to obtain a quality evaluation result output by the quality evaluation model. The quality evaluation model is obtained based on the above-mentioned model training method.

[0132] In the example quality evaluation apparatus, the pre-trained quality evaluation model is used to more accurately evaluate the quality of the depth image, and the depth image can be screened based on the quality evaluation result. The high-quality depth image is fused with the optical image to reduce the noise generated when the depth image is fused with the optical image, and the accuracy of subsequent image processing is improved.

[0133] The functions of each module can be referred to the model training method embodiments, which will not be repeated here.

[0134] Please refer to Figure 11 , Figure 11 is a structural schematic diagram of an electronic device according to an example embodiment of the present application. The electronic device 1100 includes a memory 1101 and a processor 1102, and the processor 1102 is configured to execute program instructions stored in the memory 1101 to implement the steps in any of the above-mentioned model training method embodiments. In one specific implementation scenario, the electronic device 1100 can include but is not limited to a microcomputer, a server, and in addition, the electronic device 1100 can also include a notebook computer, a tablet computer, and the like, without limitation.

[0135] Specifically, the processor 1102 is configured to control itself and the memory 1101 to implement the steps in any of the above model training method embodiments or quality evaluation method embodiments. The processor 1102 can also be referred to as a CPU (Central Processing Unit). The processor 1102 can be an integrated circuit chip having a processing capability. The processor 1102 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 1102 can be implemented by an integrated circuit chip together.

[0136] Please refer to Figure 12 , Figure 12 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 1200 stores program instructions 1210 capable of being executed by a processor, and the program instructions 1210 are used to implement the steps in any of the above model training method embodiments or quality evaluation method embodiments.

[0137] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be repeated here.

[0138] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be mutually referred to. For brevity, they will not be repeated here.

[0139] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the above-described apparatus implementation is only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual elements can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0140] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit. When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the method of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a sample optical image and a sample depth image associated with the sample optical image, the sample optical image containing a target region corresponding to a target object; performing region segmentation on the target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image; calculating the similarity between the actual region and the target region, respectively mapping the actual region and the target region to a reference image, obtaining the overlapping area between the actual region and the target region in the reference image, the merged area, calculating the ratio of the overlapping area and the merged area, obtaining a similarity constraint parameter, based on the weights corresponding to the similarity and the similarity constraint parameter, respectively, performing weighted calculation on the similarity and the similarity constraint parameter, obtaining the association score between the target region and the actual region, taking the association score as the quality label corresponding to the sample depth image; training a quality evaluation model to be trained based on the sample depth image and the quality label corresponding to the sample depth image, obtaining a trained quality evaluation model.

2. The method of claim 1, wherein, The region segmentation on the target object contained in the sample depth image to obtain the actual region corresponding to the target object contained in the sample depth image comprises: obtaining distribution information of depth values contained in the sample depth image; based on the distribution information of the depth values, calculating a target segmentation threshold corresponding to the target object contained in the sample depth image; based on the target segmentation threshold, performing region segmentation on the target object contained in the sample depth image to obtain the actual region.

3. The method of claim 2, wherein, The calculation of the target segmentation threshold corresponding to the target object contained in the sample depth image based on the distribution information of the depth values comprises: based on the distribution information of the depth values, calculating the probability corresponding to each depth value; based on the probability corresponding to each depth value, constructing an entropy value function corresponding to the sample depth image; based on the entropy value function, calculating the target segmentation threshold.

4. A method of quality assessment characterized by, The method comprises: obtaining a depth image to be evaluated; inputting the depth image to be evaluated into a pre-trained quality evaluation model to obtain a quality evaluation result output by the quality evaluation model; wherein the quality evaluation model is obtained based on the model training method in any one of claims 1 to 3.

5. A model training apparatus characterized by comprising: comprise: a sample acquisition module configured to obtain a sample optical image and a sample depth image associated with the sample optical image, the sample optical image containing a target region corresponding to a target object; a region segmentation module configured to perform region segmentation on the target object contained in the sample depth image to obtain an actual region corresponding to the target object contained in the sample depth image; The label determination module is configured to calculate a similarity between the actual region and the target region, map the actual region and the target region to a reference image respectively, obtain an overlapping area and a merged area between the actual region and the target region in the reference image, calculate a ratio of the overlapping area and the merged area to obtain a similarity constraint parameter, perform weighted calculation on the similarity and the similarity constraint parameter based on corresponding weights of the similarity and the similarity constraint parameter respectively, obtain a correlation score between the target region and the actual region, and take the correlation score as a quality label corresponding to the sample depth image. The model training module is configured to train a to-be-trained quality evaluation model based on the sample depth image and the quality label corresponding to the sample depth image to obtain a trained quality evaluation model.

6. A quality evaluation device characterized by comprising: The method comprises the following steps: The image acquisition module is configured to acquire a to-be-evaluated depth image. The quality evaluation module is configured to input the to-be-evaluated depth image into a pre-trained quality evaluation model to obtain a quality evaluation result output by the quality evaluation model, wherein the quality evaluation model is obtained based on the model training method in any one of claims 1 to 3.

7. An electronic device, comprising: The device comprises a memory and a processor, and the processor is configured to execute program instructions stored in the memory to implement the model training method in any one of claims 1 to 3 or the quality evaluation method in claim 4.

Images