Model training, image processing method and device, electronic equipment and storage medium

Abstract

The application discloses a model training and image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a sample image; converting the sample image into point cloud data to obtain first point cloud data; performing perspective transformation processing on the first point cloud data to obtain second point cloud data corresponding to the first point cloud data after perspective transformation processing; training a to-be-trained model according to the first point cloud data and the second point cloud data to obtain a trained model. Two-dimensional sample image data is converted into point cloud data to obtain the first point cloud data, and perspective transformation processing is performed on the first point cloud data to obtain the second point cloud data corresponding to the first point cloud data after perspective transformation processing. The to-be-trained model is trained according to the first point cloud data and the second point cloud data which can more accurately reflect the structure and shape characteristics of an object to obtain the trained model, so that the trained model has better feature extraction capability for the structure and shape characteristics.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a model training, image processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, machine learning has played a significant role in advancing the field of computer vision. To better accomplish downstream tasks, a large amount of sample image data is typically used to train machine learning models. This allows trained models with strong feature extraction capabilities to effectively extract features from the image data for downstream tasks.

[0003] In existing technologies, a large number of sample images are typically used to train the model to obtain the trained model. However, since the object structure and shape features included in two-dimensional images are very limited, the model trained based on two-dimensional images is inefficient at extracting the three-dimensional structure and shape features of objects, resulting in low accuracy of visual recognition. Summary of the Invention

[0004] This application provides a model training, image processing method, apparatus, electronic device, and computer-readable storage medium to address the inefficiency of extracting three-dimensional structure and shape features of objects from models trained based on two-dimensional images.

[0005] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0006] Acquire sample images;

[0007] The sample image is converted into point cloud data to obtain the first point cloud data;

[0008] The first point cloud data is subjected to a view transformation process to obtain the second point cloud data after the view transformation process.

[0009] The model to be trained is trained based on the first point cloud data and the second point cloud data to obtain the trained model.

[0010] Secondly, embodiments of this application provide an image processing method, the method comprising:

[0011] Acquire the image to be processed and convert it into point cloud data;

[0012] The point cloud data is input into an image processing model for processing to obtain the image features corresponding to the image to be processed; the image processing model is trained by the model training method described in the first aspect above.

[0013] Thirdly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0014] The first acquisition module is used to acquire sample images;

[0015] The conversion module is used to convert the sample image into point cloud data to obtain the first point cloud data;

[0016] The first processing module is used to perform view transformation processing on the first point cloud data to obtain the second point cloud data after view transformation processing corresponding to the first point cloud data.

[0017] The training module is used to train the model to be trained based on the first point cloud data and the second point cloud data to obtain the trained model.

[0018] Fourthly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0019] The second acquisition module is used to acquire the image to be processed and convert the image to be processed into point cloud data;

[0020] The second processing module is used to input the point cloud data into the image processing model for processing to obtain the image features corresponding to the image to be processed; the image processing model is trained by the model training method described in the first aspect above.

[0021] Fifthly, embodiments of this application provide an electronic device, the electronic device comprising:

[0022] A memory and a processor, wherein the memory and the processor are coupled;

[0023] The memory is used to store one or more computer instructions;

[0024] The processor is used to execute one or more computer instructions to implement the model training method described in any of the first aspects above, or to implement the image processing method described in any of the second aspects above.

[0025] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing one or more computer instructions, characterized in that the instructions are executed by a processor to implement the model training method described in any of the first aspects above, or to implement the image processing method described in any of the second aspects above.

[0026] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the model training method described in any of the first aspects above, or implements the image processing method described in any of the second aspects above.

[0027] Compared with the prior art, this application has the following advantages:

[0028] The model training method provided in this application involves: acquiring sample images; converting the sample images into point cloud data to obtain first point cloud data; performing viewpoint transformation processing on the first point cloud data to obtain second point cloud data corresponding to the first point cloud data; and training the model to be trained based on the first point cloud data and the second point cloud data to obtain the trained model. Point cloud data, as a type of three-dimensional data feature, can accurately reflect the structural and shape features of an object. A pre-trained model is created based on a point cloud feature extraction network, and the pre-trained model is trained using the object's point cloud data. After training, the pre-trained model can effectively extract the three-dimensional structural and shape features of the object from the corresponding point cloud data. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0030] Figure 1 A schematic flowchart illustrating the model training method provided in the first embodiment of this application;

[0031] Figure 2 A schematic diagram of the collection extraction module provided in the first embodiment of this application;

[0032] Figure 3 A schematic diagram of a point cloud feature extraction network consisting of N set extraction modules connected in series, as provided in the first embodiment of this application;

[0033] Figure 4 A schematic diagram of the feature aggregation module provided in the first embodiment of this application;

[0034] Figure 5 A schematic diagram of the point cloud feature extraction network provided in the first embodiment of this application;

[0035] Figure 6 A schematic flowchart illustrating the image processing method provided in the second embodiment of this application;

[0036] Figure 7 This is a schematic diagram illustrating the process of generating the image to be trained and fine-tuning the image processing model, as provided in the second embodiment of this application.

[0037] Figure 8 This is a schematic diagram of the structure of the model training device provided in the third embodiment of this application;

[0038] Figure 9This is a schematic diagram of the structure of the image processing apparatus provided in the fourth embodiment of this application;

[0039] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the fifth embodiment of this application.

[0040] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0041] To make the objectives, advantages, and features of this application clearer, the application will be described clearly and completely below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to provide a full understanding of this application. However, the described embodiments are only some, not all, of the embodiments of this application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0042] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance, or a specific order or sequence. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, the term "multiple" refers to two or more. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0043] To facilitate understanding of the technical solution of this application, the relevant concepts involved in this application will be introduced first.

[0044] Commonly used machine learning methods are mainly divided into supervised learning and unsupervised learning. Simply put, the distinction between supervised and unsupervised learning lies in whether the input training data has labels. If the training data has labels, it is supervised learning; if the training data is unlabeled, it is unsupervised learning.

[0045] Compared to supervised learning, the biggest advantage of unsupervised learning is that it does not require data labeling. Therefore, unsupervised learning can save significant manpower and resources. Furthermore, the features learned through unsupervised learning are more adaptive and richer. Unsupervised learning refers to learning features from unlabeled data. Because there are no labels, unsupervised learning methods are diverse, including clustering, dimensionality reduction, and more.

[0046] To give a simple example: In image classification, each image is labeled with a category (images contain a wealth of information, but labels only have one category, which cannot fully utilize the information in the image, or in other words, labels limit learning). Supervised learning trains a network to ultimately label each image with a category. However, when you want to use the network as a segmentation network, you need to readjust most of the layers in the network. Unsupervised learning, on the other hand, only needs to train the input image on some pre-tasks, and it can be used in various downstream tasks (including image classification, image segmentation, and other downstream tasks).

[0047] As a type of unsupervised learning, contrastive learning's main idea is to bring similar samples (i.e., positive samples) closer together in space and push dissimilar samples (i.e., negative samples) further apart in space. By optimizing this task, high-quality image representations can be obtained, and then these image representations can be used to improve the performance of downstream tasks.

[0048] Pre-trained models are network models that have been trained on a large amount of data in advance and are saved after training. Pre-trained models are considered the foundation for various downstream tasks under different data patterns. Specifically, they are based on large-scale data to train foundational models such as BERT, GPT-3, MAE, DALLE-E, and ChatGPT, providing reasonable parameter initialization for downstream applications. During the training process of pre-trained models using large-scale data, the information obtained from the data is essentially a weight of a multi-layer network. By extracting the weight parameters from the pre-trained model and transferring them to other network models, those other network models learn the features and knowledge of the original network.

[0049] Reinforcement learning (RL), also known as reward learning, evaluation learning, or reinforcement learning, is a paradigm and methodology of machine learning used to describe and solve problems in which an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment.

[0050] Reinforcement learning is a learning process where an agent learns through trial and error, using rewards gained from interacting with the environment to guide its behavior, with the goal of maximizing the reward. Unlike supervised learning, reinforcement learning primarily relies on reinforcement signals. Specifically, when an agent performs an action in the environment, the environment generates a reinforcement signal to evaluate the quality of that action; the environment does not directly tell the reinforcement learning system how to produce the correct action. Because the external environment provides limited information, the reinforcement learning system must learn through its own trial-and-error experience interacting with the environment. In this way, the reinforcement learning system acquires knowledge in the action-evaluation environment and improves its action plans to adapt to the environment.

[0051] The prior art involved in this application and the problems existing in the prior art are described below:

[0052] Reinforcement learning is one of the most effective methods for solving decision-making problems and has been widely applied in fields such as robotics, gaming, healthcare, and recommender systems. To improve the sample learning efficiency of reinforcement learning, pre-trained models with strong feature extraction capabilities are widely used in this field. Currently, machine learning has played a significant role in advancing computer vision. To better perform downstream tasks, a large amount of sample image data is typically used to train the model through machine learning. This allows the trained model, with its strong feature extraction capabilities, to effectively extract features from the image data for downstream tasks.

[0053] In existing technologies, a large number of sample images are typically used to train the model to obtain the trained model. However, since the object structure and shape features included in two-dimensional images are very limited, the model trained based on two-dimensional images is inefficient at extracting the three-dimensional structure and shape features of objects, resulting in low accuracy of visual recognition.

[0054] To address the aforementioned problems, this application provides a model training method, a corresponding model training apparatus, an image processing method, and a corresponding image processing apparatus, as well as an electronic device and a computer-readable storage medium capable of implementing the model training method or the image processing method. The following embodiments provide a detailed description of the above-described method, apparatus, electronic device, and computer-readable storage medium.

[0055] To make the objectives and technical solutions of this application clearer and more intuitive, the methods provided by the embodiments of this application will be described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit this application. It is understood that the following embodiments may exist independently, and the embodiments and features described below may be combined with each other where there is no conflict between the various embodiments provided in this application. For the same or similar content, it will not be repeated in different embodiments. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0056] This application provides a model training method, an image processing apparatus, an electronic device, and a computer storage medium. Specifically, the model training method and image processing method of one embodiment of this application can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer, or other terminal device. The terminal can also include a client, which can be a game application client, a browser client carrying a game program, or an instant messaging client, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0057] It should be noted that the following description of the technical solution provided in this application uses the server as the execution subject as an example. In other possible implementations, a control device located on the server can also be used as the execution subject to execute the technical solution provided in this application. The embodiments of this application do not limit the type of execution subject. Optionally, the control device can be implemented by software or by a combination of software and hardware.

[0058] Below, in conjunction with Figure 1 The model training method provided in the embodiments of this application will be described. Figure 1 This is a flowchart illustrating the model training method provided in the first embodiment of this application.

[0059] like Figure 1 As shown, the model training method includes:

[0060] S101. Obtain the sample image.

[0061] In this embodiment of the application, the sample image can be any image.

[0062] S102. Convert the sample image into point cloud data to obtain the first point cloud data.

[0063] In this embodiment of the application, after acquiring a two-dimensional sample image, the two-dimensional image is converted into three-dimensional point cloud data to obtain a set of points, which is referred to as the first point cloud data corresponding to the two-dimensional sample image.

[0064] In this embodiment of the application, the first point cloud data is a dataset, where each point (x, y, z) represents a set of X, Y, Z spatial geometric coordinates. When these points are combined, they form a first point cloud data, which is a set of data points in space representing a three-dimensional shape or a three-dimensional object.

[0065] In the embodiments of this application, the conversion method used to convert the sample image into point cloud data can refer to the methods in the prior art, and will not be described in detail here.

[0066] S103. Obtain the model to be trained.

[0067] In this embodiment, the model to be trained includes a point cloud feature extraction network, a mapping network, and a prediction network. The point cloud feature extraction network is primarily used to extract point cloud features from the input point cloud data. This network can be any network model capable of point cloud feature extraction. The mapping network is mainly used to further map the point cloud features extracted by the point cloud feature extraction network to a deeper latent space to obtain latent vectors, thereby obtaining data with discriminative power, which facilitates better learning by the initial pre-trained model.

[0068] Optionally, a contrastive training model is obtained, which is used for comparative learning by the model to be trained. This contrastive training model includes a point cloud feature extraction network and a mapping network. The main purpose of this contrastive training model is to obtain contrast metrics.

[0069] Optionally, the prediction network in the model to be trained is mainly used to map the extracted point cloud features to the latent space through the mapping network, and then pass the prediction network to obtain the final output, which serves as the prediction value of the model to be trained against the training model.

[0070] Optionally, the model to be trained can update its parameters using gradient descent.

[0071] Optionally, for each parameter in the comparison training model, the corresponding parameter is found from the parameters of the model to be trained, and the specific values ​​of the parameters in the comparison training model are updated according to the specific values ​​of the corresponding parameter. For example, in the model to be trained, the parameters in the comparison training model are compared with those in the model to be trained. The corresponding parameters are parameters For example, the parameters in the model to be trained The specific value is Compare the parameters in the training model The specific value is So, the updated parameters ,in, The attenuation rate, .

[0072] Optionally, the point cloud feature extraction network can consist of N cascaded extraction modules. The extraction module originates from the Set Abstraction module in the PointNet++ network. This module can extract point cloud features hierarchically, enabling the filtering and aggregation of point cloud features. The extraction module includes a keypoint sampling module, a local point aggregation module, and a feature extraction module. Specifically, the keypoint sampling module is mainly used to sample the input point cloud data, selecting multiple center points from the input point cloud data; the local point aggregation module is mainly used to utilize the multiple point data sampled by the keypoint sampling module, dividing the multiple point data into multiple different regions containing point data; the feature extraction module is used to encode each region containing point data obtained by the local point aggregation module to obtain the feature vector of the point cloud data. For the extraction module, the Set Abstraction module in the PointNet++ network can be referenced in existing technologies. A schematic diagram of the extraction module can be found in [reference needed]. Figure 2 , Figure 2 This is a schematic diagram of the set extraction module provided in the first embodiment of this application. N set extraction modules are connected in series to form a multi-level point cloud feature extraction network, which can be used as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of a point cloud feature extraction network consisting of N set extraction modules connected in series, as provided in the first embodiment of this application.

[0073] It is important to note that the number of concatenated ensemble extraction modules can be adjusted according to the difficulty of the task, and the parameters within each ensemble extraction module can also be set according to the actual situation. Therefore, the parameters and scale of the point cloud feature extraction network can be adjusted according to task requirements, which allows the point cloud feature extraction network to be easily embedded into various reinforcement learning frameworks.

[0074] The following describes the three main steps of the ensemble extraction module for extracting point cloud features:

[0075] Step 1: Randomly select an initial point from the point cloud data, and use the farthest point sampling method to sample a certain number of key points in the point cloud data.

[0076] Step 2: Using the sampled key points as aggregation centers, determine a sphere with a preset length threshold as its radius, and aggregate all point clouds located inside the sphere into the same group.

[0077] Step 3: Extract features from the point cloud data within each group. Input the point cloud data into the ensemble extraction module for feature extraction, and the ensemble extraction module will finally output the extracted point cloud features.

[0078] Optionally, based on the point cloud feature extraction network consisting of N cascaded extraction modules, the extraction module can be used in conjunction with a feature aggregation module. The feature aggregation module is primarily used to extract more comprehensive and stable point cloud features. The feature aggregation module can be a PointSIFT sub-module within the PointSIFT network. The three main modules constituting the feature aggregation module are a spatial segmentation module, an oriented convolution module, and a feature stacking module. A schematic diagram of the feature aggregation module can be found in [reference needed]. Figure 4 , Figure 4 This is a schematic diagram of the feature aggregation module provided in the first embodiment of this application. Specifically, the feature aggregation module can be added before any set extraction module, which enables the point cloud feature extraction network to achieve excellent scalability. Since the feature aggregation module can extract relatively comprehensive and stable features, this further enables the point cloud feature extraction network to achieve better feature extraction performance. Simultaneously, the feature aggregation module can appear in pairs with the set extraction module, or multiple set extraction modules can be paired with one feature aggregation module; this embodiment of the application does not impose any limitations on this. Combining the feature aggregation module and the set extraction module constitutes the point cloud feature extraction network. Two combination methods for combining the feature aggregation module and the set extraction module are as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of the point cloud feature extraction network provided in the first embodiment of this application. In the first combination method, a feature aggregation module is added before each set extraction module in the point cloud feature extraction network. In the second combination method, multiple set extraction modules are paired with one feature extraction module. For example, if the point cloud feature extraction network includes five set extraction modules, a feature extraction module can be added before the first set extraction module and before the third-to-last set extraction module; that is, the first feature extraction module is paired with the first two set extraction modules, and the second feature extraction module is paired with the last three feature extraction modules.

[0079] The following describes the main steps of the feature aggregation module in aggregating point cloud features:

[0080] Step 1: Divide the space into eight subspaces centered on the feature point, and find the nearest point in each subspace as the representative of that direction.

[0081] Step 2: Perform directional convolution along the x, y, and z directions respectively, that is, combine the features pairwise.

[0082] Step 3: Stack and stitch the features obtained from different directions together to obtain the final features.

[0083] In this embodiment, a feature aggregation module based on the PointSIFT network is introduced, which helps the point cloud feature extraction network to extract more comprehensive and stable point cloud features. At the same time, the feature aggregation module can be easily added or removed, which makes the point cloud feature extraction network highly scalable.

[0084] S104. Perform viewpoint change processing on the first point cloud data to obtain the second point cloud data after viewpoint change corresponding to the first point cloud data.

[0085] In this embodiment of the application, in order to simulate the first point cloud data of the target object from different perspectives, the existing first point cloud data can be processed by perspective transformation to obtain the second point cloud data corresponding to the first point cloud data after perspective transformation.

[0086] Optionally, the viewpoint transformation processing method includes at least: performing random rotation, translation, and scaling operations on a single first point cloud data, randomly combining these operations, and randomly selecting parameters for these operations.

[0087] For example, in one transformation, the first point cloud data is rotated by 10°, translated by 7% of the total scale, and scaled by 0.96. The point cloud data obtained in this way is similar to the second point cloud data obtained by photographing the target object corresponding to the first point cloud data from different perspectives and positions, and is visually different from the original first point cloud data. Augmenting the first point cloud data to obtain the second point cloud data helps the trained network to have some adaptability to the scale invariance and rotation invariance of point cloud data. Augmenting the first point cloud data yields the augmented second point cloud data.

[0088] S105. Train the model to be trained based on the first point cloud data and the second point cloud data to obtain the trained model.

[0089] Optionally, a model to be trained and a comparative training model are obtained, the comparative training model being used for comparative learning of the model to be trained; the first input data of the model to be trained is determined based on the first point cloud data and the second point cloud data, and the first input data is input into the model to be trained to obtain a predicted value; the second input data of the model to be trained is determined based on the first point cloud data and the second point cloud data, and the second input data is input into the comparative training model to obtain a target value; the parameters of the model to be trained are updated based on the preset loss function, the target value, and the predicted value to obtain the trained model.

[0090] Optionally, the specific implementation of the step "determine the first input data of the model to be trained based on the first point cloud data and the second point cloud data" can be as follows: according to a preset rule, perform a first sampling process on the synthetic point cloud data composed of the first point cloud data and the second point cloud data to obtain the first sampled point cloud data, that is, the first sampled point cloud data is the first input data.

[0091] Optionally, the specific implementation of the step "determine the second input data of the model to be trained based on the first point cloud data and the second point cloud data" can be as follows: according to a preset rule, perform a second sampling process on the synthetic point cloud data composed of the first point cloud data and the second point cloud data to obtain the second sampled point cloud data, that is, the second sampled point cloud data is the second input data.

[0092] Optionally, the preset rules include any of the following: random sampling rules, polling sampling rules, and sampling rules based on the number of samples.

[0093] Optionally, after obtaining the first and second sampled point cloud data, in order to enable the point cloud feature extraction network to better learn the spatial structure and shape features of the point cloud data, and to enhance the robustness of the point cloud feature extraction network to point cloud feature perception, spatial transformation processing is performed on the first and second sampled point cloud data respectively. Specifically, spatial transformation processing is performed on the first sampled point cloud data to obtain the spatially transformed third point cloud data corresponding to the first sampled point cloud data. And spatial transformation processing is performed on the second sampled point cloud data to obtain the spatially transformed fourth point cloud data corresponding to the second sampled point cloud data.

[0094] Optionally, spatial transformation processing includes at least one of the following data processing methods: cropping, cutting, dithering, discarding, downsampling, and normalization.

[0095] The following section provides a detailed explanation of the various data processing methods involved in the spatial transformation process described above.

[0096] The first method is cropping. This method uses a random three-dimensional cuboid to crop the point cloud data, retaining the point cloud data located inside the cuboid. The percentage of the point cloud volume occupied by the three-dimensional cuboid can be set according to actual needs, and this application embodiment does not impose any restrictions on this.

[0097] The second method is segmentation, which also uses a random three-dimensional cuboid to segment the point cloud data. Unlike random cropping, random segmentation discards the point cloud data within the three-dimensional cuboid and retains the point cloud data outside the cuboid. The percentage of the point cloud volume relative to the volume of the three-dimensional cuboid can be set according to actual needs; this application's embodiments do not impose any limitations.

[0098] The third method is jitter, which causes the three-dimensional position of each point in the point cloud data to shift randomly by a certain distance within a certain range.

[0099] The fourth method is discarding, which randomly discards a preset percentage of 3D points in the point cloud data. The preset percentage is less than 100%, and its specific value can be set according to actual needs; this application's embodiments do not impose any limitations.

[0100] The fifth method is downsampling. This method downsamples the point cloud data based on the input size of the point cloud feature extraction network and randomly selects the required number of 3D points.

[0101] The sixth method is normalization, which normalizes the point cloud scale (coordinate values) to fit a unit sphere.

[0102] Optionally, considering that random cropping and random cutting have a significant impact on point cloud data, using these two processing methods can yield point cloud data that differs more from the original point cloud data.

[0103] In the embodiments of this application, the above-mentioned methods can be randomly combined to perform different spatial transformation processes on two different sampled data to obtain the final two point cloud data.

[0104] Optionally, after obtaining the first sampled point cloud data and the second sampled point cloud data, the first sampled point cloud data and the second sampled point cloud data are respectively input into the model to be trained and the comparison training model to obtain the predicted value and the target value, respectively. That is, the first sampled point cloud data is input into the model to be trained to obtain the predicted value, and the second sampled point cloud data is input into the comparison training model to obtain the target value.

[0105] Optionally, for both the model to be trained and the comparison training model, a max pooling network is added between the point cloud feature extraction network and the mapping network. The max pooling network is used to extract the point with the largest value within the local receptive domain, thus preserving the original features while reducing the number of training parameters and shortening the training time. Taking the first sampled point cloud data as an example, the first sampled point cloud data is input into the model to be trained. The point cloud feature extraction network extracts point cloud features from the first sampled point cloud data. After this part of the point cloud features is input into the max pooling network, the point cloud features output by the max pooling network are determined as the final obtained point cloud features, and these final obtained point cloud features are input into the mapping network.

[0106] Optionally, the mapping network employs a multilayer perceptron with activation functions and batch normalization. The main function of the mapping network is to further map the extracted point cloud features to a deeper latent space, thereby obtaining discriminative data. This discriminative data is then used to train the model to be trained, enabling the model to learn point cloud features more effectively.

[0107] Optionally, a prediction network can be added after the mapping network of the model to be trained. The vector mapped to the latent space is then passed through the prediction network to obtain the final output, which is used as the prediction value for the target network.

[0108] Alternatively, the prediction network can employ a multilayer perceptron with activation functions and batch normalization.

[0109] The following section details the specific implementation of the step "Calculate the loss value based on the preset loss function, target value, and predicted value, update the parameters of the model to be trained, and obtain the pre-trained model."

[0110] Optionally, a loss value is calculated based on a preset loss function, the target value, and the predicted value. The parameters of the model to be trained are updated based on this loss value to obtain a pre-trained model. The loss value is calculated on the target value and the predicted value using the preset loss function. The model to be trained is then trained based on this loss value, enabling it to correctly learn the features of the point cloud data. The trained model is then identified as the pre-trained model. The preset loss function can be a mean squared error loss function.

[0111] Optionally, the preset loss function can be the mean squared error loss function. Specifically, the mean squared error is the expected value of the squared difference between the target value and the predicted value. During training, the mean squared loss value between the target value and the predicted value should be minimized as much as possible, that is, the closer the predicted value is to the target value, the better the prediction effect.

[0112] Optionally, the parameters of the model to be trained can be updated using gradient descent.

[0113] After training, the model to be trained is identified as the pre-trained model.

[0114] The model training method provided in this application includes: acquiring sample images; converting the sample images into point cloud data to obtain first point cloud data; acquiring a model to be trained and a comparison model; performing viewpoint transformation processing on the first point cloud data to obtain second point cloud data corresponding to the first point cloud data after viewpoint transformation processing; and training the model to be trained based on the first point cloud data and the second point cloud data to obtain a trained model. Specifically, the two-dimensional sample image data is converted into point cloud data to obtain the first point cloud data. To obtain point cloud data from other viewpoints, the first point cloud data undergoes viewpoint transformation processing to obtain second point cloud data corresponding to the first point cloud data. The first and second point cloud data can reflect the structural and shape features of an object in greater detail. Training the model to be trained based on the first and second point cloud data yields a trained model, which enables the trained model to have better feature extraction capabilities for the three-dimensional structure and shape features of an object.

[0115] In the embodiments of this application, the model training method is applicable to different task scenarios and task requirements. It does not require a downstream reinforcement learning task with explicit feature orientation and can easily achieve good point cloud feature extraction results when the size of the point cloud feature extraction network is different.

[0116] Below, in conjunction with Figure 6 The image processing method provided in the embodiments of this application will be described. Figure 6 This is a schematic flowchart of the image processing method provided in the second embodiment of this application.

[0117] like Figure 6 As shown, the image processing method includes:

[0118] S601. Obtain the image to be processed and convert it into point cloud data.

[0119] In the embodiments of this application, the description and explanation of converting the image to be processed into point cloud data can refer to the description and explanation of step S102 above, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0120] S602. Input the point cloud data into the image processing model for processing to obtain the image features corresponding to the image to be processed.

[0121] In this embodiment, after converting the image to be processed in the downstream task into point cloud data, the image processing model trained by the model training method provided in the above embodiment is used to extract point cloud features from the point cloud data after the point cloud data corresponding to the image to be processed, which can extract more three-dimensional structure and shape features of the object.

[0122] In existing technologies, most reinforcement learning pre-training methods have not solved the problem of mismatch between pre-trained models and downstream reinforcement learning tasks. That is, the data distribution of downstream tasks is not consistent with the training data used for pre-training, and the difference can be quite large.

[0123] Furthermore, in order to solve the above problems, based on the above embodiments, step S602 may optionally include steps S6021-6024:

[0124] S6021. After obtaining the training images for the image processing model and downstream tasks, the image processing model is trained using the model training method provided in the above embodiments.

[0125] S6022. Convert the image to be trained into point cloud data to obtain the point cloud data to be trained.

[0126] S6023. Fine-tune the image processing model based on the point cloud data to be trained to obtain the downstream task processing model.

[0127] S6024. The image to be processed is processed based on the downstream task processing model to obtain the image features corresponding to the image to be processed.

[0128] Below, in conjunction with Figure 7 The following examples illustrate the specific implementation of fine-tuning the image processing model using the point cloud data of the training image converted from the training image of the downstream task.

[0129] Images related to downstream tasks are generated and used as training images for those tasks. These training images are then converted into point cloud data to obtain the training point cloud data. The image processing model is then fine-tuned using this training point cloud data. Figure 7 This is a schematic diagram illustrating the process of generating the image to be trained and fine-tuning the image processing model, as provided in the second embodiment of this application. Figure 7 As shown, a random strategy is used to allow the agent to interact with the environment, and the depth map of the environment obtained during the interaction is used as the training image for the downstream task. The depth map is converted into point cloud data using virtual camera intrinsics to obtain the training point cloud data. Then, the image processing model is trained based on the training point cloud data obtained from the downstream task to fine-tune the image processing model, which enables the fine-tuned image processing model to better perform the downstream task.

[0130] Optionally, the image processing model can be further trained with a small learning rate using the point cloud data to be trained, so as to adjust the parameters of the image processing model and make the adjusted image processing model more suitable for downstream reinforcement learning tasks.

[0131] In the embodiments of this application, a depth map refers to an image that uses the distance values ​​of each point in the scene acquired by the image acquisition device as pixel values, and it directly reflects the geometry of the visible surface of the object.

[0132] Furthermore, considering that existing point cloud-based reinforcement learning grasping networks suffer from low sample learning efficiency and slow network convergence speed, this paper addresses these issues. Sample learning efficiency refers to the amount of data required for the learning system to reach any chosen target performance level. High sample learning efficiency requires less data, while low efficiency requires more. It can also be understood as the proportion of experience (or gain) gained from learning from a sample relative to the maximum gain of that sample during the learning process.

[0133] To address the aforementioned issues, the image processing model trained using the model training method provided in the above embodiments is applied to a reinforcement learning grasping network. This image processing model effectively extracts point cloud features from the environmental point cloud data and performs dimensionality reduction on these features to obtain the environmental state variables. The dimensionality reduction methods for point cloud features can include any of the following: multilayer perceptron, max pooling network, etc. During the object grasping process, to obtain a complete environmental state variable, the reinforcement learning grasping network concatenates the environmental state variable with the robotic arm's own state variable, using the concatenated result as the complete environmental state variable. Subsequently, this environmental state variable is input into the reinforcement learning grasping network, and the network is trained based on the environmental state variables of the downstream task.

[0134] Optionally, an environment state extraction network based on a multilayer perceptron or a max pooling layer is designed. This environment state extraction network is used to perform dimensionality reduction on point cloud features to obtain an environment state quantity that can be applied to reinforcement learning grasping networks. This provides an important guarantee for communication between image processing models and reinforcement learning networks based on downstream tasks.

[0135] Existing techniques typically freeze the parameters of image processing models during the training process of reinforcement learning networks based on image processing models. This limits the crawling performance of such networks to some extent. In other words, the parameters of the image processing model are not updated during the training of the reinforcement learning crawling network.

[0136] Optionally, during the training of the reinforcement learning network, backpropagation can be used to fine-tune the parameters of the image processing model. This can significantly optimize the grasping performance of the reinforcement learning grasping network based on the image processing model. It should be noted that due to the high randomness of reinforcement learning, the point cloud feature extraction network matched with each training strategy is different. Fixing the parameters of the image processing model is equivalent to ignoring the differences between each training strategy. However, if a large learning rate is used to update the parameters of the image processing model, it will lead to the loss of the model's existing knowledge. To avoid the loss of existing knowledge, a small learning rate is used to update the parameters of the image processing model. This ensures that the model's existing knowledge is not lost while matching the reinforcement learning strategy as closely as possible, thereby significantly optimizing the grasping performance of the reinforcement learning grasping network based on the image processing model.

[0137] The model training apparatus provided in this application is described below. The model training apparatus described below can be referred to in correspondence with the model training method described above.

[0138] Figure 8 This is a schematic diagram of the model training device provided in the third embodiment of this application. Figure 8 As shown, the model training device includes: an acquisition module 801, a conversion module 802, a processing module 803, and a training module 804.

[0139] The first acquisition module 801 is used to acquire sample images;

[0140] The conversion module 802 is used to convert the sample image into point cloud data to obtain the first point cloud data;

[0141] The first processing module 803 is used to perform view transformation processing on the first point cloud data to obtain the second point cloud data after view transformation processing corresponding to the first point cloud data.

[0142] The training module 804 is used to train the model to be trained based on the first point cloud data and the second point cloud data to obtain the trained model.

[0143] Optionally, the training module 804 is specifically used for:

[0144] Obtain a model to be trained and a comparative training model, wherein the comparative training model is used for the model to be trained to perform comparative learning;

[0145] The first input data of the model to be trained is determined based on the first point cloud data and the second point cloud data, and the first input data is input into the model to be trained to obtain the predicted value.

[0146] The second input data of the model to be trained is determined based on the first point cloud data and the second point cloud data, and the second input data is input into the comparison training model to obtain the target value;

[0147] Based on the preset loss function, the target value, and the predicted value, the parameters of the model to be trained are updated to obtain the trained model.

[0148] Optionally, the training module 804 is specifically used for:

[0149] According to preset rules, the synthetic point cloud data composed of the first point cloud data and the second point cloud data is subjected to a first sampling process to obtain the first sampled point cloud data.

[0150] The step of determining the second input data of the model to be trained based on the first point cloud data and the second point cloud data includes:

[0151] The synthesized point cloud data is subjected to a second sampling process to obtain second sampled point cloud data.

[0152] Optionally, the training module 804 is specifically used for:

[0153] The loss value is determined based on a preset loss function, the target value, and the predicted value.

[0154] Based on the loss value, the parameters of the model to be trained are updated to obtain the trained model.

[0155] Optionally, the training module 804 is specifically used for:

[0156] The first sampled point cloud data is subjected to spatial transformation processing to obtain the third point cloud data after spatial transformation processing corresponding to the first sampled point cloud data.

[0157] The first sampled point cloud data and the third point cloud data are input into the model to be trained to obtain the predicted value;

[0158] The step of inputting the second input data into the contrastive training model to obtain the target value includes:

[0159] The second sampled point cloud data is subjected to spatial transformation processing to obtain the fourth point cloud data after spatial transformation processing corresponding to the second sampled point cloud data.

[0160] The second sampled point cloud data and the fourth point cloud data are input into the comparison training model to obtain the target value.

[0161] Optionally, the viewpoint transformation processing includes at least one of the following data processing methods: rotation, translation, and scaling.

[0162] Optionally, the spatial transformation process includes at least one of the following data processing methods: cropping, cutting, dithering, discarding, downsampling, and normalization.

[0163] Optionally, the preset rules include any one of the following: random sampling rules, polling sampling rules, and sampling rules based on the number of samples.

[0164] The model training device provided in this embodiment can be used to execute the technical solution of the above-described model training method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0165] The image processing apparatus provided in this application is described below. The image processing apparatus described below can be referred to in correspondence with the image processing method described above.

[0166] Figure 9 This is a schematic diagram of the image processing apparatus provided in the fourth embodiment of this application. Figure 9 As shown, the image processing device includes: a first acquisition module 901 and a second processing module 902.

[0167] The second acquisition module 901 is used to acquire the image to be processed and convert the image to be processed into point cloud data;

[0168] The second processing module 902 is used to input the point cloud data into the image processing model for processing to obtain the image features corresponding to the image to be processed; the image processing model is trained by performing any of the model training methods described above.

[0169] Optionally, the second processing module 902 is further configured to:

[0170] The image processing model is obtained by training the image processing model and the downstream task by performing any of the model training methods described above.

[0171] The image to be trained is converted into point cloud data to obtain the point cloud data to be trained;

[0172] The image processing model is fine-tuned based on the point cloud data to be trained to obtain the downstream task processing model;

[0173] The image to be processed is processed based on the downstream task processing model to obtain the image features corresponding to the image to be processed.

[0174] The image processing apparatus provided in this embodiment can be used to execute the technical solutions of the above-described image processing method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0175] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the fifth embodiment of this application, as shown below. Figure 10 As shown, the electronic device 1000 of this embodiment includes: a processor 1001 and a memory 1002; wherein

[0176] Memory 1002 is used to store computer-executed instructions;

[0177] Processor 1001 is configured to execute computer execution instructions stored in memory to perform the model training method or the image processing method provided by the above-described methods. The model training method includes: acquiring a sample image; converting the sample image into point cloud data to obtain first point cloud data; performing viewpoint transformation processing on the first point cloud data to obtain second point cloud data corresponding to the first point cloud data after viewpoint transformation processing; and training a model to be trained based on the first point cloud data and the second point cloud data to obtain a trained model. The image processing method includes: acquiring an image to be processed and converting the image to be processed into point cloud data; secondly, inputting the point cloud data into an image processing model for processing to obtain image features corresponding to the image to be processed; the image processing model is trained using the model training methods provided by the above-described methods.

[0178] Alternatively, the memory 1002 can be either standalone or integrated with the processor 1001.

[0179] When the memory 1002 is set up independently, the electronic device also includes a bus 1003 for connecting the memory 1002 and the processor 1001.

[0180] The sixth embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the model training method or the image processing method provided by the above-mentioned electronic device.

[0181] The seventh embodiment of this application also provides a computer program product, the program product comprising: a computer program, the computer program being stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, the at least one processor executing the computer program causing the electronic device to perform the model training method provided by the above methods, or the image processing method provided by the above methods.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0183] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or a processor to execute some steps of the methods described in the various embodiments of this application.

[0184] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0185] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0186] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0187] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0188] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method includes: Acquire sample images; The sample image is converted into point cloud data to obtain the first point cloud data; The first point cloud data is subjected to a view transformation process to obtain the second point cloud data after the view transformation process. The training model is obtained by training the model to be trained based on the first point cloud data and the second point cloud data. The step of training the model to be trained based on the first point cloud data and the second point cloud data to obtain the trained model includes: Obtain a model to be trained and a comparative training model, wherein the comparative training model is used for the model to be trained to perform comparative learning; The first input data of the model to be trained is determined based on the first point cloud data and the second point cloud data, and the first input data is input into the model to be trained to obtain the predicted value. The second input data of the model to be trained is determined based on the first point cloud data and the second point cloud data, and the second input data is input into the comparison training model to obtain the target value; Based on the preset loss function, the target value, and the predicted value, the parameters of the model to be trained are updated to obtain the trained model. The step of determining the first input data of the model to be trained based on the first point cloud data and the second point cloud data includes: performing a first sampling process on the synthetic point cloud data composed of the first point cloud data and the second point cloud data according to a preset rule to obtain the first sampled point cloud data. The step of determining the second input data of the model to be trained based on the first point cloud data and the second point cloud data includes: performing a second sampling process on the synthetic point cloud data to obtain second sampled point cloud data; The step of inputting the first input data into the model to be trained to obtain a predicted value includes: performing spatial transformation processing on the first sampled point cloud data to obtain spatially transformed third point cloud data corresponding to the first sampled point cloud data; and inputting the first sampled point cloud data and the third point cloud data into the model to be trained to obtain a predicted value. The step of inputting the second input data into the comparison training model to obtain the target value includes: performing spatial transformation processing on the second sampled point cloud data to obtain the spatially transformed fourth point cloud data corresponding to the second sampled point cloud data; and inputting the second sampled point cloud data and the fourth point cloud data into the comparison training model to obtain the target value.

2. The method according to claim 1, characterized in that, The step of updating the parameters of the model to be trained based on a preset loss function, the target value, and the predicted value to obtain the trained model includes: The loss value is determined based on the preset loss function, the target value, and the predicted value; Based on the loss value, the parameters of the model to be trained are updated to obtain the trained model.

3. The method according to claim 1, characterized in that, The perspective transformation processing includes at least one of the following data processing methods: rotation, translation, and scaling.

4. The method according to claim 1, characterized in that, The spatial transformation process includes at least one of the following data processing methods: cropping, cutting, dithering, discarding, downsampling, and normalization.

5. The method according to claim 1, characterized in that, The preset rules include any one of the following: random sampling rules, polling sampling rules, and sampling rules based on the number of samples.

6. An image processing method, characterized in that, include: Acquire the image to be processed and convert it into point cloud data; The point cloud data is input into an image processing model for processing to obtain the image features corresponding to the image to be processed; the image processing model is trained by the model training method described in any one of claims 1-5.

7. The method according to claim 6, characterized in that, The step of inputting the point cloud data into an image processing model for processing to obtain the image features corresponding to the image to be processed includes: Obtain the training images for the image processing model and downstream tasks; The image to be trained is converted into point cloud data to obtain the point cloud data to be trained; The image processing model is fine-tuned based on the point cloud data to be trained to obtain the downstream task processing model; The image to be processed is processed based on the downstream task processing model to obtain the image features corresponding to the image to be processed.

8. A model training device, characterized in that, The device includes: The first acquisition module is used to acquire sample images; The conversion module is used to convert the sample image into point cloud data to obtain the first point cloud data; The first processing module is used to perform view transformation processing on the first point cloud data to obtain the second point cloud data after view transformation processing corresponding to the first point cloud data. The training module is used to train the model to be trained based on the first point cloud data and the second point cloud data to obtain the trained model. Specifically, the training module is used to: acquire a model to be trained and a comparative training model, wherein the comparative training model is used for the model to be trained to perform comparative learning; determine the first input data of the model to be trained based on the first point cloud data and the second point cloud data, and input the first input data into the model to be trained to obtain a predicted value; determine the second input data of the model to be trained based on the first point cloud data and the second point cloud data, and input the second input data into the comparative training model to obtain a target value; and update the parameters of the model to be trained based on a preset loss function, the target value, and the predicted value to obtain the trained model. The training module is further specifically used to: perform a first sampling process on the synthetic point cloud data composed of the first point cloud data and the second point cloud data according to a preset rule to obtain first sampled point cloud data; and perform a second sampling process on the synthetic point cloud data to obtain second sampled point cloud data. The training module is further specifically used for: performing spatial transformation processing on the first sampled point cloud data to obtain spatially transformed third point cloud data corresponding to the first sampled point cloud data; inputting the first sampled point cloud data and the third point cloud data into the model to be trained to obtain a predicted value; performing spatial transformation processing on the second sampled point cloud data to obtain spatially transformed fourth point cloud data corresponding to the second sampled point cloud data; and inputting the second sampled point cloud data and the fourth point cloud data into the comparison training model to obtain a target value.

9. An image processing apparatus, characterized in that, include: The second acquisition module is used to acquire the image to be processed and convert the image to be processed into point cloud data; The second processing module is used to input the point cloud data into the image processing model for processing to obtain the image features corresponding to the image to be processed; the image processing model is trained by the model training method according to any one of claims 1-5.

10. An electronic device, characterized in that, The electronic device includes: Processor; and A memory for storing a data processing program, which, when powered on and run by the processor, executes the model training method as described in any one of claims 1-5, or the image processing method as described in claim 6 or 7.

11. A computer-readable storage medium, characterized in that, The system stores a data processing program that is executed by a processor to perform the model training method as described in any one of claims 1-5, or the image processing method as described in claim 6 or 7.

Images