Image augmentation method and device, electronic equipment and readable storage medium
Through the improved target neural radiation field and text-image pre-training model, combined with multimodal information, a large amount of labeled data is generated, which solves the accuracy and efficiency problems of image data augmentation in the prior art, and realizes the video image data Effectively expand.
Patent Information
- Application Number
- CN202310572603.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art cannot accurately and effectively perform image data augmentation, especially in generating pixel-level, accurate labeled data.
通过获取目标图像的信息,输入改进后的目标神经辐射场,利用三维图像信息生成不同视角的二维图像,并构建损失函数以优化神经辐射场,结合文本-图像预训练模型,实现多模态的图像生成架构,生成大量标注数据。
Accurate augmentation of image data is achieved, multimodal image data sets are generated, sample imbalance problem is solved, and video image data is improved.
Smart Images

Figure CN119007203A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the technical field of video services, and in particular to a method, device, electronic device, and readable storage medium for image augmentation. Background Art
[0002] Existing technology typically augments spatial object data through 3D reconstruction, using multi-viewpoint imagery to reconstruct a point cloud to obtain a 3D model of the object. This model is then rendered into a 2D image using commonly used rendering software. This method relies on third-party rendering software and cannot generate datasets for image segmentation or instance segmentation, nor can it generate accurate pixel-level annotations. Therefore, there is a need to address the problem of how to use a small amount of data collected and annotated to synthesize a large amount of annotated data through image generation to achieve data augmentation. Summary of the Invention
[0003] Embodiments of the present invention provide a method, apparatus, electronic device, and readable storage medium for image augmentation to solve the existing problem of being unable to accurately and effectively augment image data.
[0004] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:
[0005] In a first aspect, an embodiment of the present invention provides a method for image data augmentation, comprising:
[0006] Acquire information of a target image, where the target image is a two-dimensional image, and the information of the target image includes semantic labels corresponding to each segmented region in the target image;
[0007] Inputting information of the target image into a target neural radiation field, and outputting information of a three-dimensional image corresponding to the target image, wherein the information of the three-dimensional image includes different viewing angles of each segmented area of the target image;
[0008] Information of a plurality of two-dimensional images corresponding to different perspectives in the target image is obtained according to the information of the three-dimensional image, wherein the information of the two-dimensional image includes semantic labels corresponding to each region in the segmented two-dimensional image.
[0009] Optionally, the information of the target image further includes at least one of the following: an RGB color image of the target image and a mask image with semantic labels corresponding to various regions in the target image.
[0010] Optionally, the information of the two-dimensional image further includes: an RGB color image of the segmented two-dimensional image and a mask image with semantic labels corresponding to each region in the segmented two-dimensional image.
[0011] Optionally, after obtaining information of a plurality of two-dimensional images corresponding to different perspectives in the target image according to information of the three-dimensional image, the method further includes:
[0012] Constructing a first loss function based on information of a plurality of two-dimensional images corresponding to different perspectives of each region in the target image;
[0013] Setting weights for the first loss function according to semantic labels corresponding to respective regions in the segmented two-dimensional image, to obtain a weighted first loss function, wherein the weights of semantic labels corresponding to regions with a large pixel ratio in the segmented two-dimensional image are greater than the weights of semantic labels corresponding to regions with a small pixel ratio in the segmented two-dimensional image;
[0014] The parameters of the target neural radiation field are adjusted according to the first loss function after setting the weights, and the target neural radiation field is optimized.
[0015] Optionally, after obtaining information of a plurality of two-dimensional images corresponding to different perspectives in the target image according to information of the three-dimensional image, the method further includes:
[0016] Inputting the RGB color map of the segmented two-dimensional image and the semantic labels corresponding to the respective regions in the segmented two-dimensional image into a text-image pre-training model to obtain a first eigenvector of the RGB color map and a second eigenvector of the semantic labels;
[0017] Obtaining a second loss function of a cosine distance between the first eigenvector and the second eigenvector according to the first eigenvector and the second eigenvector;
[0018] The parameters of the target neural radiation field are adjusted according to the second loss function to optimize the target neural radiation field.
[0019] Optionally, also include:
[0020] Obtaining all target pixels with the same semantic label in the semantic labels corresponding to each region in the segmented two-dimensional image;
[0021] The minimum bounding rectangle of the target pixel is obtained to obtain a detection data set of the target pixel.
[0022] In a second aspect, an embodiment of the present invention provides an apparatus for image data augmentation, comprising:
[0023] A first acquisition module is configured to acquire information of a target image, wherein the target image is a two-dimensional image, and the information of the target image includes semantic labels corresponding to each segmented region in the target image;
[0024] a first processing module, configured to input information of the target image into a target neural radiation field, and output information of a three-dimensional image corresponding to the target image, wherein the information of the three-dimensional image includes different viewing angles of each segmented area of the target image;
[0025] The second processing module is used to obtain information of multiple two-dimensional images corresponding to different perspectives in the target image based on the information of the three-dimensional image, where the information of the two-dimensional image includes semantic labels corresponding to each area in the segmented two-dimensional image.
[0026] Optionally, the information of the target image further includes at least one of the following: an RGB color image of the target image and a mask image with semantic labels corresponding to various regions in the target image.
[0027] Optionally, the information of the two-dimensional image further includes: an RGB color image of the segmented two-dimensional image and a mask image with semantic labels corresponding to each region in the segmented two-dimensional image.
[0028] Optionally, also include:
[0029] A third processing module is configured to construct a first loss function based on information of a plurality of two-dimensional images corresponding to different viewing angles of each region in the target image;
[0030] a fourth processing module, configured to weight the first loss function according to the semantic labels corresponding to the respective regions in the segmented two-dimensional image, to obtain a weighted first loss function, wherein the weights of the semantic labels corresponding to the regions with a large pixel ratio in the segmented two-dimensional image are greater than the weights of the semantic labels corresponding to the regions with a small pixel ratio in the segmented two-dimensional image;
[0031] The first optimization module is used to adjust the parameters of the target neural radiation field according to the first loss function after setting the weights, so as to optimize the target neural radiation field.
[0032] Optionally, also include:
[0033] a fifth processing module, configured to input the RGB color map of the segmented two-dimensional image and the semantic labels corresponding to the respective regions in the segmented two-dimensional image into a text-image pre-training model to obtain a first eigenvector of the RGB color map and a second eigenvector of the semantic labels;
[0034] a sixth processing module, configured to obtain, based on the first eigenvector and the second eigenvector, a second loss function of a cosine distance between the first eigenvector and the second eigenvector;
[0035] The second optimization module is used to adjust the parameters of the target neural radiation field according to the second loss function to optimize the target neural radiation field.
[0036] Optionally, also include:
[0037] A second acquisition module is used to acquire all target pixels with the same semantic label in the semantic labels corresponding to each region in the segmented two-dimensional image;
[0038] The seventh processing module is used to obtain the minimum bounding rectangle of the target pixel to obtain a detection data set of the target pixel.
[0039] In a third aspect, an embodiment of the present invention provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps in the method for image data augmentation as described in any one of the first aspects.
[0040] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the method for image data augmentation as described in any one of the first aspects are implemented.
[0041] In the present invention, by obtaining the information of the target image and inputting the information of the target image into the improved target neural radiation field with added semantic information, a small amount of data is collected and labeled, and then a large amount of labeled data is synthesized by means of image generation. This solves the existing problem of being unable to accurately and effectively augment image data, and forms a multimodal image generation architecture. By utilizing multimodal information, the augmentation of video image data is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:
[0043] Figure 1 is a flowchart of a method for image data augmentation provided by an embodiment of the present invention;
[0044] Figure 2 1 is a schematic diagram of the structure of a neural network of an image data augmentation method provided by an embodiment of the present invention;
[0045] Figure 3 This is a contrast loss flow chart of a method for image data augmentation provided by an embodiment of the present invention;
[0046] Figure 4 1 is a schematic diagram of the overall structure of a method for image data augmentation provided by an embodiment of the present invention;
[0047] Figure 5 1 is a schematic structural diagram of an apparatus for image data augmentation provided by an embodiment of the present invention;
[0048] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0050] Please refer to Figure 1 , an embodiment of the present invention provides a method for image data augmentation, comprising:
[0051] Step 11: Acquire information of a target image, where the target image is a two-dimensional image. The information of the target image includes semantic labels corresponding to each segmented region in the target image.
[0052] In an embodiment of the present invention, the target image is the collected original image or the image obtained by extracting frames from the collected video. The target image is annotated according to the image segmentation annotation method to obtain the target image information. The semantic label includes semantic text information and segmentation area information. The target image information also includes at least one of the following: an RGB color image of the target image and a mask image with semantic labels corresponding to each area in the target image. By introducing semantic information, the subsequent improved target neural radiation field is facilitated to process data according to the semantic labels, thereby facilitating the construction of a multimodal architecture.
[0053] Step 12: Inputting the target image information into the target neural radiation field, and outputting the information of the three-dimensional image corresponding to the target image, wherein the three-dimensional image information includes different viewing angles of each segmented area of the target image;
[0054] In the embodiment of the present invention, please refer to Figure 2The target neural radiation field is an improved neural radiation field with semantic information. The target neural radiation field represents a three-dimensional scene. In order to introduce semantic information, a semantic segmentation branch is added to the target neural radiation field. The semantic labels of each part of the model in the three-dimensional space of the target neural radiation field are only related to the position and have nothing to do with the observation viewpoint direction. By improving the target neural radiation field, the image data can be effectively augmented and expanded.
[0055] The input of the target neural radiation field is the position coordinates (x, y, z) and the viewpoint direction (θ, φ), and the output is color information (r, g, b), three-dimensional volume density σ and semantic label s. The three-dimensional scene is implicitly represented by a multi-layer perceptron (MLP). The positional encoding (Positional Encoding) operation in the target neural radiation field first positionally encodes the position coordinates (x, y, z) and the viewpoint direction (θ, φ) and then inputs the MLP. The color information is related to both the position coordinates and the viewpoint direction, and the three-dimensional volume density is only related to the position.
[0056] Step 13: obtaining information of multiple two-dimensional images corresponding to different perspectives in the target image based on the information of the three-dimensional image, wherein the information of the two-dimensional image includes semantic labels corresponding to each region in the segmented two-dimensional image.
[0057] In an embodiment of the present invention, optionally, the information of the two-dimensional image also includes: an RGB color map of the segmented two-dimensional image and a mask image with semantic labels corresponding to each area in the segmented two-dimensional image; by rendering the target neural radiation field, a two-dimensional image is obtained, a small amount of data is collected and labeled, and then a large amount of labeled data is synthesized by means of image generation, thereby realizing the augmentation of image data.
[0058] In an embodiment of the present invention, by obtaining information of a target image and inputting the information of the target image into an improved target neural radiation field with added semantic information, a small amount of data is collected and labeled, and then a large amount of labeled data is synthesized by means of image generation. This solves the existing problem of being unable to accurately and effectively augment image data, and forms a multimodal image generation architecture. By utilizing multimodal information, the augmentation of video image data is achieved.
[0059] In the embodiment of the present invention, optionally, after obtaining information of a plurality of two-dimensional images corresponding to different perspectives in the target image according to information of the three-dimensional image, the method further includes:
[0060] Constructing a first loss function based on information of a plurality of two-dimensional images corresponding to different perspectives of each region in the target image;
[0061] Setting weights for the first loss function according to semantic labels corresponding to respective regions in the segmented two-dimensional image, to obtain a weighted first loss function, wherein the weights of semantic labels corresponding to regions with a large pixel ratio in the segmented two-dimensional image are greater than the weights of semantic labels corresponding to regions with a small pixel ratio in the segmented two-dimensional image;
[0062] The parameters of the target neural radiation field are adjusted according to the first loss function after setting the weights, and the target neural radiation field is optimized.
[0063] In an embodiment of the present invention, a segmentation training loss function of semantic information is set to alleviate the problem of imbalance in the semantic label categories of the segmentation, and different weights ω are set for different semantic labels in the segmentation to solve the problem of sample imbalance. The parameters of the target neural radiation field are adjusted according to the first loss function after setting the weight, and the target neural radiation field is optimized.
[0064] For example: increase the weight of foreground pixels and reduce the weight of background pixels; different categories are assigned weights according to the proportion of pixels they occupy, with smaller proportions having larger weights and larger proportions having smaller weights.
[0065] The first loss function is the segmentation training loss function of semantic information, and the formula is as follows:
[0066] Among them, ω is the weight of different semantic labels in segmentation; r is each starting from the camera center
[0067]
[0068] Light ray; R is all sample rays in a training batch; l is a category of semantic label; L is all categories of semantic label; p l (r), and are the segmentation probability of the true value under the semantic label l category, the probability of the coarse volume density estimation (output of the coarse network), and the probability of the fine volume density estimation (output of the fine network).
[0069] In the embodiment of the present invention, optionally, after obtaining information of a plurality of two-dimensional images corresponding to different perspectives in the target image according to information of the three-dimensional image, the method further includes:
[0070] Inputting the RGB color map of the segmented two-dimensional image and the semantic labels corresponding to the respective regions in the segmented two-dimensional image into a text-image pre-training model to obtain a first eigenvector of the RGB color map and a second eigenvector of the semantic labels;
[0071] Obtaining a second loss function of a cosine distance between the first eigenvector and the second eigenvector according to the first eigenvector and the second eigenvector;
[0072] The parameters of the target neural radiation field are adjusted according to the second loss function to optimize the target neural radiation field.
[0073] In the embodiment of the present invention, when rendering the generated image, the RGB color map of the segmented two-dimensional image and the semantic labels corresponding to each region in the segmented two-dimensional image are input into the text-image pre-training model, and the first feature vector x is extracted by the image encoder. image At the same time, the semantic label text is passed through the text encoder to extract the second feature vector x text , using this as supervision, constructing a second loss function of the cosine distance between the first eigenvector and the second eigenvector, adjusting the parameters of the target neural radiation field according to the second loss function, and optimizing the target neural radiation field;
[0074] The second loss function is a training loss function for text-image similarity, and the formula is as follows:
[0075]
[0076] Among them, margin is a parameter that can be adjusted to change the distance scale of similarity.
[0077] In the embodiment of the present invention, information of multiple two-dimensional images corresponding to different perspectives in the target image is obtained based on the information of the three-dimensional image, that is, the required pixel-level segmentation annotation map is generated and produced by rendering. The specific process is as follows:
[0078] Using the formula:
[0079]
[0080] The trained three-dimensional implicit representation model is rendered into two-dimensional image information, namely, an RGB color map of the segmented two-dimensional image and a mask image with semantic labels corresponding to each region in the segmented two-dimensional image.
[0081] Among them, ω k is the distance weight; t k There are K sampling points on a ray, and the current position is k; r is the ray connecting the camera center to a pixel of the 2D segmentation mask; The semantic label predicted for a pixel in the two-dimensional segmentation mask annotation image can then be converted into the probability of multi-class semantics through a normalization layer; is the current sampling point t k The cumulative opacity at is:
[0082]
[0083] σ(t k ) is the current sampling point t k The volume density at δ k =t k+1 -t k is the distance between two adjacent orthogonal sampling points; s(t k ) is the current sampling point t k Semantic tags at
[0084] Among them, the distance weight
[0085] ω k The semantics of the segmentation category closer to the lens contributes more to the semantics of the final segmentation region.
[0086] In an embodiment of the present invention, the text-image pre-training model needs to be trained and optimized before use. The RGB color image of the target image in the collected data set, that is, the information of the target image, and the RGB image and the text of the semantic label in the ground truth of the mask image with the semantic labels corresponding to each area in the target image are connected. First, feature extraction is performed on the image and text respectively. Among them, the backbone of image feature extraction can adopt the Resnet residual network series model or the VIT series model, and the text feature extraction can adopt the Bert algorithm model. After feature extraction, feature vectors are obtained respectively, and the two feature vectors are vector-multiplied to calculate the cosine distance. The result of the same matching pair is close to 1, and the result of different matching pairs is close to 0. The model is optimized using the InfoNCE Loss loss function based on the results of the matching pairs.
[0087] Please refer to Figure 3 , InfoNCE Loss loss function is based on sampling method, which converts multi-classification problem into binary classification problem, where x query is the sample to be checked, q is the feature after being encoded by the encoder, x key is a sample that can be found in the dictionary, k is the sample feature after being encoded by the encoder, that is, the key in the dictionary, and the contrast loss is calculated between q and a group of k, where the calculation formula of the contrast loss is Among them, q and k can also be expressed in other forms, such as similarity metrics and cosine similarity; the numerator represents the similarity between positive examples, and the denominator represents the similarity between positive and negative examples. That is, the greater the similarity within the same category, the smaller the similarity between different categories, and the smaller the loss. After training is completed, the text-image pre-training model is integrated into the first loss function and the second loss function described above. The total loss function is the sum of the losses of the semantic information segmentation training loss function, the text-image similarity training loss function, and the color loss function. This allows for joint optimization, achieving an accurate augmented dataset, and effectively helping to improve the accuracy of the training model.
[0088] Please refer to Figure 4 In an embodiment of the present invention, a three-dimensional scene is represented by using a target neural radiation field with semantic information, and the true value ground truth is used as supervision. Through the text-image and training model, the RGB image and the corresponding semantic label in the true value groundtruth are respectively passed through the encoder, and their matching relationship is learned through comparative learning. After the learning is completed, the true value semantic label is used as supervision, and the text image similarity loss function is used to constrain the optimization process, and the loss function is optimized to update the parameters in the neural radiation field to achieve the purpose of optimization. After the optimization is completed, the obtained improved target neural radiation field is rendered to obtain a two-dimensional image, including an RGB image and a segmentation mask image with semantic labels, that is, information of multiple two-dimensional images corresponding to different perspectives in the target image.
[0089] In an embodiment of the present invention, a three-dimensional vision-based method is designed to utilize an improved target neural radiation field with added semantic information, and a multimodal image generation architecture utilizing text information, integrating differentiable rendering and comparative text image pre-training models, and designing a training loss that includes a loss function of semantic information and a loss function of text image similarity to help train the joint model; at the same time, by designing a rendering method that outputs pixel-level segmentation annotations, multimodal information is utilized to achieve video image data augmentation.
[0090] In the embodiment of the present invention, optionally, the method further includes:
[0091] Obtaining all target pixels with the same semantic label in the semantic labels corresponding to each region in the segmented two-dimensional image;
[0092] The minimum bounding rectangle of the target pixel is obtained to obtain a detection data set of the target pixel.
[0093] In an embodiment of the present invention, in addition to generating a segmented dataset, an instance segmentation dataset and a target detection dataset can also be generated. By obtaining the rendered RGB image and its corresponding pixel-level semantic segmentation, the minimum enclosing rectangle of all target pixels with the same semantic label in the semantic labels corresponding to each area in the segmented two-dimensional image is calculated to obtain the target detection box, that is, the detection dataset of the target pixels.
[0094] Please refer to Figure 5 , an embodiment of the present invention provides a device for image data augmentation, comprising:
[0095] A first acquisition module 51 is configured to acquire information of a target image, where the target image is a two-dimensional image and the information of the target image includes semantic labels corresponding to each segmented region in the target image;
[0096] A first processing module 52 is configured to input information of the target image into a target neural radiation field and output information of a three-dimensional image corresponding to the target image, wherein the information of the three-dimensional image includes different viewing angles of each segmented area of the target image;
[0097] The second processing module 53 is configured to obtain information of multiple two-dimensional images corresponding to different perspectives in the target image based on the information of the three-dimensional image, where the information of the two-dimensional image includes semantic labels corresponding to various regions in the segmented two-dimensional image.
[0098] In the embodiment of the present invention, optionally, the information of the target image further includes at least one of the following: an RGB color image of the target image and a mask image with semantic labels corresponding to various regions in the target image.
[0099] In the embodiment of the present invention, optionally, the information of the two-dimensional image further includes: an RGB color image of the segmented two-dimensional image and a mask image with semantic labels corresponding to each region in the segmented two-dimensional image.
[0100] In the embodiment of the present invention, optionally, the method further includes:
[0101] A third processing module is configured to construct a first loss function based on information of a plurality of two-dimensional images corresponding to different viewing angles of each region in the target image;
[0102] a fourth processing module, configured to weight the first loss function according to the semantic labels corresponding to the respective regions in the segmented two-dimensional image, to obtain a weighted first loss function, wherein the weights of the semantic labels corresponding to the regions with a large pixel ratio in the segmented two-dimensional image are greater than the weights of the semantic labels corresponding to the regions with a small pixel ratio in the segmented two-dimensional image;
[0103] The first optimization module is used to adjust the parameters of the target neural radiation field according to the first loss function after setting the weights, so as to optimize the target neural radiation field.
[0104] In the embodiment of the present invention, optionally, the method further includes:
[0105] a fifth processing module, configured to input the RGB color map of the segmented two-dimensional image and the semantic labels corresponding to the respective regions in the segmented two-dimensional image into a text-image pre-training model to obtain a first eigenvector of the RGB color map and a second eigenvector of the semantic labels;
[0106] a sixth processing module, configured to obtain, based on the first eigenvector and the second eigenvector, a second loss function of a cosine distance between the first eigenvector and the second eigenvector;
[0107] The second optimization module is used to adjust the parameters of the target neural radiation field according to the second loss function to optimize the target neural radiation field.
[0108] In the embodiment of the present invention, optionally, the method further includes:
[0109] A second acquisition module is used to acquire all target pixels with the same semantic label in the semantic labels corresponding to each region in the segmented two-dimensional image;
[0110] The seventh processing module is used to obtain the minimum bounding rectangle of the target pixel to obtain a detection data set of the target pixel.
[0111] The image data augmentation device provided by the embodiment of the present invention can achieve Figure 1 The various processes implemented by the method embodiment achieve the same technical effect and are not described here again to avoid repetition.
[0112] The embodiment of the present invention provides an electronic device 60, see Figure 6 As shown, Figure 6 This is a principle block diagram of an electronic device 60 according to an embodiment of the present invention, which includes a processor 61, a memory 62, and a program or instruction stored in the memory 62 and executable on the processor 61. When the program or instruction is executed by the processor, the steps of any one of the image data augmentation methods according to the present invention are implemented.
[0113] An embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the embodiment of the method for image data augmentation as described above are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.
[0114] Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.
[0115] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.
[0116] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.
[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a service classification device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0118] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for image data augmentation, characterized in that: include: Acquire information of a target image, wherein the target image is a two-dimensional image, and the information of the target image includes semantic labels corresponding to each segmented area in the target image; Inputting information of the target image into a target neural radiation field, and outputting information of a three-dimensional image corresponding to the target image, wherein the information of the three-dimensional image includes different viewing angles of each segmented area of the target image; Information of a plurality of two-dimensional images corresponding to different viewing angles in the target image is obtained according to the information of the three-dimensional image, wherein the information of the two-dimensional image includes semantic labels corresponding to each region in the segmented two-dimensional image.
2. The method for image data augmentation according to claim 1, characterized in that: The information of the target image also includes at least one of the following: an RGB color image of the target image and a mask image with semantic labels corresponding to various regions in the target image.
3. The method for image data augmentation according to claim 1, characterized in that: The information of the two-dimensional image also includes: an RGB color image of the segmented two-dimensional image and a mask image with semantic labels corresponding to each area in the segmented two-dimensional image.
4. The method for image data augmentation according to claim 3, characterized in that: After obtaining information of a plurality of two-dimensional images corresponding to different viewing angles in the target image according to the information of the three-dimensional image, the method further includes: Constructing a first loss function based on information of a plurality of two-dimensional images corresponding to different viewing angles of each region in the target image; Setting weights for the first loss function according to the semantic labels corresponding to the respective regions in the segmented two-dimensional image, to obtain a weighted first loss function, wherein the weights of the semantic labels corresponding to the regions with a large pixel ratio in the segmented two-dimensional image are greater than the weights of the semantic labels corresponding to the regions with a small pixel ratio in the segmented two-dimensional image; The parameters of the target neural radiation field are adjusted according to the first loss function after setting the weights, so as to optimize the target neural radiation field.
5. The method for image data augmentation according to claim 3, characterized in that: After obtaining information of a plurality of two-dimensional images corresponding to different viewing angles in the target image according to the information of the three-dimensional image, the method further includes: Inputting the RGB color map of the segmented two-dimensional image and the semantic labels corresponding to each region in the segmented two-dimensional image into a text-image pre-training model to obtain a first feature vector of the RGB color map and a second feature vector of the semantic label; Obtaining a second loss function of a cosine distance between the first eigenvector and the second eigenvector according to the first eigenvector and the second eigenvector; The parameters of the target neural radiation field are adjusted according to the second loss function to optimize the target neural radiation field.
6. The method for image data augmentation according to claim 1, characterized in that: Also includes: Acquire all target pixels with the same semantic label in the semantic labels corresponding to each region in the segmented two-dimensional image; The minimum bounding rectangle of the target pixel is obtained to obtain a detection data set of the target pixel.
7. A device for image data augmentation, characterized in that: include: A first acquisition module is used to acquire information of a target image, where the target image is a two-dimensional image, and the information of the target image includes semantic labels corresponding to each segmented area in the target image; A first processing module, configured to input information of the target image into a target neural radiation field, and output information of a three-dimensional image corresponding to the target image, wherein the information of the three-dimensional image includes different viewing angles of each segmented area of the target image; The second processing module is used to obtain information of multiple two-dimensional images corresponding to different perspectives in the target image according to the information of the three-dimensional image, and the information of the two-dimensional image includes semantic labels corresponding to each area in the segmented two-dimensional image.
8. The apparatus for image data augmentation according to claim 7, characterized in that: The information of the target image also includes at least one of the following: an RGB color image of the target image and a mask image with semantic labels corresponding to various regions in the target image.
9. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps in the method for image data augmentation according to any one of claims 1 to 6.
10. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps in the method for image data augmentation according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Semantic segmentation method and device, model training method and device, equipment and storage medium
CN113920314A
Three-dimensional scene graph generation method and device, equipment and readable storage medium
CN115393386A
Image processing method and device, electronic equipment and storage medium
CN115880526A
High-fidelity three-dimensional face model generation method based on natural text description
CN115984485A
