Multi-view image data generation method and device, terminal equipment and storage medium

Abstract

The invention discloses a multi-view image data generation method and device, terminal equipment and a storage medium, and belongs to the technical field of image processing. The multi-view image data generation method comprises the following steps: acquiring initial view image data; and performing multi-style processing on the initial view angle image data through an adversarial generative network model to obtain multi-view angle image data corresponding to the initial view angle image data. By adopting the technical scheme of the invention, the technical effect of reducing the acquisition difficulty of the trained data set can be realized, and the deep neural network model with a better effect can be obtained through training.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, terminal device and storage medium for generating multi-view image data. Background Art

[0002] With the continuous development of deep neural networks, deep neural networks are increasingly being used in scenarios such as self-driving cars, cancer detection, and complex games. In related technologies, a large number of labeled data sets are required for training to obtain deep neural networks. However, it is very difficult to obtain labeled data sets. Therefore, when it is impossible to obtain a large number of data sets, related technologies only use small sample data sets to train deep neural network models. This will make it difficult for the trained deep neural network models to achieve satisfactory recognition effects during application. Therefore, how to reduce the difficulty of obtaining data sets has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0003] The main purpose of this application is to provide a method, device, terminal device and storage medium for generating multi-view image data, aiming to reduce the difficulty of acquiring data sets, thereby enabling training to obtain a better deep neural network model.

[0004] To achieve the above object, an embodiment of the present application provides a method for generating multi-view image data, the method comprising:

[0005] Acquire initial viewing angle image data;

[0006] The initial perspective image data is processed in multiple styles through an adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data.

[0007] In addition, an embodiment of the present application further provides a device for generating multi-view image data, the device comprising:

[0008] An image acquisition module, configured to acquire preset initial viewing angle image data;

[0009] The image processing module is configured to perform multi-style processing on the initial perspective image data through a preset adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data.

[0010] In addition, an embodiment of the present application also provides a terminal device, which includes: a multi-view image data generation device as described above, a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multi-view image data generation method as described above.

[0011] In addition, an embodiment of the present application further provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for generating multi-view image data as described above are implemented.

[0012] The embodiments of the present application disclose a method, apparatus, terminal device and storage medium for generating multi-perspective image data, including: obtaining initial perspective image data; performing multi-style processing on the initial perspective image data through an adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data.

[0013] Therefore, the embodiment of the present application obtains initial perspective image data for training, and inputs the initial perspective image data into the trained adversarial generative network model, and the adversarial generative network model performs multi-style processing on the initial perspective image data to generate multi-perspective image data with different perspectives.

[0014] In this way, the present application performs multi-style processing on the image data used for training through an adversarial generative network model, and then generates image data of other perspectives corresponding to the image data, so that in the process of training the deep neural network model, the small sample data set used for training can be expanded to obtain more data sets for training, thereby achieving the technical effect of reducing the difficulty of obtaining data sets, and then being able to train a deep neural network model with better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the structure of a terminal device in the hardware operating environment involved in the embodiment of the present application;

[0016] Figure 2 A schematic diagram of the steps of the first embodiment of the method for generating multi-view image data provided by the embodiment of the present application;

[0017] Figure 3 A schematic diagram of the steps of a preferred embodiment of a method for generating multi-view image data provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of an image processing flow involved in an embodiment of a method for generating multi-view image data provided in an embodiment of the present application;

[0019] Figure 5 A vehicle perspective conversion effect diagram according to an embodiment of a method for generating multi-perspective image data provided in an embodiment of the present application;

[0020] Figure 6This is a diagram of experimental results of a traditional detection network model involved in an embodiment of a method for generating multi-view image data provided in an embodiment of the present application;

[0021] Figure 7 This is a diagram of experimental results of a target detection network involved in an embodiment of a method for generating multi-view image data provided in an embodiment of the present application;

[0022] Figure 8 A schematic diagram of the functional modules involved in the device for providing multi-view image data according to the embodiment of the present application.

[0023] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] Reference Figure 1 , Figure 1 This is a schematic diagram of the terminal device structure of the hardware operating environment involved in the embodiment of the present application.

[0026] In this embodiment, the terminal device of the hardware operating environment involved in the embodiment of the present application may be a terminal device that is internally configured with a model training unit, can perform model training through the model training unit, and can call the trained detection model. Alternatively, the terminal device involved in the embodiment of the present application may also be a data storage control terminal, mobile terminal, PC or other terminal connected to the above-mentioned model training unit.

[0027] like Figure 1As shown, the terminal device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0028] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the terminal device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0029] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module and a computer program.

[0030] exist Figure 1 In the terminal device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the terminal device of the present application can be set in the terminal device, and the terminal device calls the computer program stored in the memory 1005 through the processor 1001, and performs the following operations:

[0031] Acquire initial viewing angle image data;

[0032] Performing multi-style processing on the initial perspective image data through an adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data;

[0033] Exemplarily, the processor 1001 may call a computer program stored in the memory 1005, and further perform the following operations:

[0034] Segmenting the initial perspective image data through an adversarial generative network model to obtain a target area and a background area corresponding to the initial perspective image data;

[0035] Generate mask vectors corresponding to the target area and the background area respectively;

[0036] Multi-style processing is performed on the initial view image data based on the mask vector and the style control vector to obtain multi-view image data.

[0037] Exemplarily, the processor 1001 may call a computer program stored in the memory 1005, and further perform the following operations:

[0038] Performing multi-style processing on the initial perspective image data based on the mask vector and the style control vector to generate corresponding virtual images and real images;

[0039] determining authenticity between the virtual image and the real image;

[0040] Based on the authenticity, a generator in the adversarial generative network model is updated to obtain a target generator;

[0041] The initial view image data is processed based on the target generator to obtain multi-view image data.

[0042] Exemplarily, the processor 1001 may call a computer program stored in the memory 1005, and further perform the following operations:

[0043] Acquire a plurality of initial image data, and process the plurality of initial image data respectively to generate a plurality of enhanced image data;

[0044] Integrate the target image data in the plurality of the initial image data and the target enhanced data corresponding to the target image data in the plurality of the enhanced image data to obtain an initial image data set;

[0045] The adversarial generative network model is obtained by training based on the initial image data set.

[0046] Exemplarily, the processor 1001 may call a computer program stored in the memory 1005, and further perform the following operations:

[0047] Annotating the multi-view image data to obtain sample data annotation;

[0048] A target detection network model is obtained based on the sample data annotation and the multi-view image data training, so as to process the real-time environment image data through the target detection network model to obtain a model prediction result.

[0049] Exemplarily, the processor 1001 may call a computer program stored in the memory 1005, and further perform the following operations:

[0050] Acquire real-time environment image data, and process the real-time environment image data through the target detection network model to obtain image feature information corresponding to the real-time environment image data;

[0051] A model prediction result is obtained based on the image feature information.

[0052] Exemplarily, the processor 1001 may call a computer program stored in the memory 1005, and further perform the following operations:

[0053] Get the preset validation data set;

[0054] The target detection network model is verified based on the verification data set to obtain an average model accuracy corresponding to the target detection network model.

[0055] Based on the terminal device structure of the hardware operating environment involved in the above-mentioned embodiment of the present application, the overall concept of the method for generating multi-view image data provided by the embodiment of the present application is first proposed.

[0056] In the process of training deep neural networks, related technologies require the use of a large number of labeled data sets for training to obtain deep neural networks. However, it is very difficult to obtain labeled data sets. Therefore, related technologies only use small sample data sets to train deep neural network models when it is impossible to obtain a large number of data sets, which will make it difficult for the trained deep neural network models to achieve satisfactory recognition effects during application.

[0057] In response to the above phenomenon, the embodiment of the present application proposes a method for generating multi-perspective image data, the method comprising: obtaining initial perspective image data; performing multi-style processing on the initial perspective image data through an adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data.

[0058] In this way, compared with the difficulty in obtaining training data sets in related technologies, the embodiments of the present application can expand the small sample data sets used for training to obtain more data sets for training during the training of the deep neural network model, thereby achieving the technical effect of reducing the difficulty of obtaining data sets, and then being able to train a better deep neural network model.

[0059] Based on the overall concept of the method for generating multi-view image data provided by the above-mentioned embodiment scheme of the present application, various embodiments of the method for generating multi-view image data of the present application are proposed.

[0060] It should be noted that the method for generating multi-view image data of the present application is applied to the above-mentioned terminal device, which may be a terminal device that is internally configured with a model training unit, can perform model training through the model training unit, and can call the detection model obtained by training. It should be understood that based on different design needs of actual applications, in different feasible embodiments, the method for generating multi-view image data of the present application can of course also be applied to other terminal devices, and in order to facilitate understanding and explanation of the technical solution, the following text will use the terminal device with an internally integrated model training unit as the executor of the solution implementation to explain the method for generating multi-view image data of the present application.

[0061] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the steps of the first embodiment of the method for generating multi-view image data of the present application. It should be noted that although the logical order is shown in the flowchart, in some cases, the method for generating multi-view image data of the present application can of course perform the steps shown or described in a different order than that here.

[0062] like Figure 2 As shown, in the first embodiment of the method for generating multi-view image data of the present application, the method for generating multi-view image data provided by the present application may include the following steps:

[0063] Step S10: Acquire initial viewing angle image data;

[0064] In this embodiment, when the user needs to train the target detection network model, the user first inputs the initial viewing angle image data containing a single shooting viewing angle for training into the terminal device.

[0065] It should be noted that, in this embodiment and another embodiment, the initial perspective image data is image data containing a single perspective acquired by the technician through an image acquisition device before model training, and the number of samples corresponding to the initial perspective image data may be smaller than the number of samples required corresponding to the model training process.

[0066] Step S20: performing multi-style processing on the initial perspective image data through an adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data;

[0067] In this embodiment, the terminal device inputs the acquired initial perspective image data into an internally configured model training unit, and the model training unit inputs the initial perspective image data into a trained adversarial generative network model, so that the adversarial generative network performs sample annotation and processing on the input initial perspective image data to obtain multi-perspective image data corresponding to the initial perspective image data.

[0068] For example, see Figure 5 , Figure 5 The vehicle perspective conversion effect diagram involved in the method for generating multi-perspective image data provided by the embodiment of the present application is a vehicle perspective conversion effect diagram. The terminal device inputs the initial perspective image data used to train the target detection network model into the internally configured model training unit, and the model training unit inputs the initial perspective image data into the adversarial generative network model. The adversarial generative network model then performs sample annotation and multi-style processing on the initial perspective image through the internal semantic segmentation network and target detection network, thereby generating Figure 5 The multi-perspective image data shown corresponds to the initial perspective image data and has multiple image styles and multiple shooting perspectives.

[0069] In this way, the present application performs multi-style processing on the image data used for training through an adversarial generative network model, and then generates image data of other perspectives corresponding to the image data, so that in the process of training the deep neural network model, the small sample data set used for training can be expanded to obtain more data sets for training, thereby achieving the technical effect of reducing the difficulty of obtaining data sets, and then being able to train a deep neural network model with better results.

[0070] Optionally, in some feasible embodiments of the method for generating multi-view image data of the present application, the operation of performing multi-style processing on the initial view image data through the adversarial generative network model to obtain multi-view image data corresponding to the initial view image data may include:

[0071] Segmenting the initial perspective image data through an adversarial generative network model to obtain a target area and a background area corresponding to the initial perspective image data;

[0072] Generate mask vectors corresponding to the target area and the background area respectively;

[0073] Performing multi-style processing on the initial view image data based on the mask vector and the style control vector to obtain multi-view image data;

[0074] In this embodiment, when the model training unit inputs the initial perspective image data into the adversarial generative network model, the adversarial generative network model calls the internal semantic segmentation network to segment the initial perspective image data into corresponding target areas and background areas, and the adversarial generative network then calls the target detection network to generate corresponding mask vectors for the target area and the background area, respectively, thereby performing multi-style processing on the initial perspective image data based on each mask vector and a preset style control vector to generate multi-perspective image data with multiple image styles and multiple shooting perspectives corresponding to the initial perspective image data.

[0075] Exemplarily, for example, when the model training unit inputs the initial perspective image data into the adversarial generative network model, the adversarial generative network model inputs the initial perspective image data input by the user into the internal semantic segmentation network, and the semantic segmentation network uses the semantic segmentation function to cut the passable area part and the background area part contained in the initial perspective image data to obtain the target area and background area corresponding to the initial perspective image data. The adversarial generative network then calls the target detection network to generate mask vectors corresponding to the target area and the background area respectively, and inputs the mask vector into the mapping network layer. The mapping network layer processes the initial perspective image data based on the mask vector and the style control vector selected by the technician to generate multi-perspective image data corresponding to the initial perspective image data and having multiple image styles and multiple shooting perspectives.

[0076] In this way, the adversarial generative network model can obtain the background area and target area corresponding to the initial perspective image data by cutting the input initial perspective image data, and then generate mask vectors corresponding to the background area and the target area respectively, and process the initial perspective image data based on the mask vector and the style control vector to generate multi-perspective image data with multiple image styles and multiple shooting perspectives corresponding to the initial perspective image data, thereby achieving the purpose of expanding the image data set used for training and reducing the difficulty of obtaining the image data set.

[0077] Optionally, in some feasible embodiments of the method for generating multi-view image data of the present application, the operation of performing multi-style processing on the initial view image data based on the mask vector and the style control vector to obtain the multi-view image data may include:

[0078] Performing multi-style processing on the initial perspective image data based on the mask vector and the style control vector to generate corresponding virtual images and real images;

[0079] determining authenticity between the virtual image and the real image;

[0080] Based on the authenticity, a generator in the adversarial generative network model is updated to obtain a target generator;

[0081] Processing the initial view image data based on the target generator to obtain multi-view image data;

[0082] In this embodiment, the mapping network layer of the adversarial generative network model inputs the input initial angle image data into the internal generator, and the generator enhances the initial angle image data to obtain target perspective enhanced image data. The generator processes the initial perspective image data to generate virtual initial perspective image data, and processes the target perspective enhanced image data to generate virtual perspective enhanced image data. The adversarial generative network model then calls the internal discriminator to calculate the authenticity of the initial angle image data, the target perspective enhanced image data, the virtual initial perspective image data and the virtual perspective enhanced image data, and updates the generator based on the authenticity to obtain the overall loss value, and when it is determined that the overall loss value is less than the preset loss value threshold, the target generator is trained, and the adversarial generative network model then calls the target generator to process the initial perspective image data to obtain multi-perspective image data corresponding to the initial perspective image data and having multiple image styles and multiple shooting perspectives.

[0083] For example, the mapping network layer of the adversarial generative network model enhances the input initial perspective image data to obtain the target perspective enhanced image data. At the same time, the mapping layer network defines the initial image as domain A and locates the enhanced image in domain B. The mapping layer network then inputs the initial perspective image data into the generator G A2B The initial perspective image data is converted into virtual perspective enhanced image data. At the same time, the mapping layer network inputs the target perspective enhanced image data into the generator G B2A The virtual initial view image data is generated in the mapping layer network, and then the discriminators DA and DB in the mapping layer network respectively determine the initial view image data and the target view enhanced image data as and The virtual perspective enhanced image data and the virtual initial perspective image data are determined as and Each discriminator thus and Mapped back to the initial view image data to obtain and Finally, when the generator is processing image data, the generator G A2B Will As input, we learn the mapping relationship between domain A and domain B to generate the corresponding At the same time, the generator G B2A The generated Mapping back to the initial view image data generation Thus we get and The corresponding training processes are:

[0084]

[0085] and

[0086]

[0087] The mapping layer network updates the generator based on the above training process and determines whether the image data output by the generator is a real image or a virtual image to obtain the image authenticity corresponding to the image data output by the generator. The mapping layer network calculates the overall loss value of the generative adversarial network based on the image authenticity:

[0088] L(G1,G2,D1,D2)=L GAN (G1,G2,X,Y)+L GAN (D1,D2,X,Y)+λL cyc (G1,G2);

[0089] Among them, L GAN To counter the loss function, it is expressed as:

[0090]

[0091] Similarly, L cyc is the cycle consistency function, which is expressed as:

[0092]

[0093] Finally, the mapping network layer compares the calculated overall loss value with the preset loss value threshold, and when it is determined that the overall loss value is less than the loss value threshold, it determines that the generator training is completed to obtain the target generator, and inputs the initial perspective image data into the target generator, so that the target generator processes the initial angle image to generate the target angle image data.

[0094] In this way, the mapping network layer within the adversarial generative network model can identify the image data output by the generator through the built-in discriminator, and then train the generator based on the identification result to generate a target generator, so that the initial perspective image data is processed by the target generator to generate more realistic multi-perspective image data.

[0095] Optionally, in some feasible embodiments of the method for generating multi-view image data of the present application, in addition to invoking the adversarial generative network model to expand the initial view image data to obtain the multi-view image data, the terminal device can also use each initial image data to train the adversarial generative network model. That is, before the above-mentioned operation of "obtaining the initial view image data", the method for generating multi-view image data of the present application can also include:

[0096] Acquire a plurality of initial image data, and process the plurality of initial image data respectively to generate a plurality of enhanced image data;

[0097] Integrate the target image data in the plurality of the initial image data and the target enhanced data corresponding to the target image data in the plurality of the enhanced image data to obtain an initial image data set;

[0098] The adversarial generative network model is obtained by training based on the initial image data set.

[0099] In this embodiment, before expanding the initial perspective image data, the terminal device needs to obtain each initial image data used to train the adversarial generative network model, and input each initial image data into the model training unit, and the model training unit performs enhancement processing on each initial image data to obtain each enhanced image data. The model training unit then integrates the target image data in each initial image data and the target enhanced image data corresponding to the target image data in each enhanced image data to obtain an initial image data set, and performs model training based on the initial image data set to obtain an adversarial generative network model.

[0100] Exemplarily, for example, before expanding the initial perspective image data used to train the target detection network model, the terminal device first obtains the initial image data and inputs the initial image data into the model training unit. The model training unit performs data enhancement such as inversion, translation, rotation, scaling, and cropping on the initial image data to obtain a first enhanced image. After that, the model training unit performs color transformation enhancement such as adding Gaussian noise, adding salt and pepper noise, adjusting brightness, adjusting saturation, adjusting contrast, color inversion, histogram equalization, and color balance on the initial image data and the first enhanced image to obtain a second enhanced image. After that, the model training unit performs Cutout, Mixup, One-hot, Cutmix, Random The third enhanced image is obtained by erasing and other advanced data enhancements. The model training unit then integrates the first enhanced images, the second enhanced images and the third enhanced images into enhanced image data, and determines the target initial image data that need to be merged in the initial image data, and determines the target enhanced image data corresponding to the target initial image data in each enhanced image data. The model training unit then integrates the target initial image data and the target enhanced image data to generate an initial image data set, and performs model training based on the initial image data set to obtain an adversarial generative network model.

[0101] In this way, the model training unit can obtain an initial image data set containing more elements by performing enhancement operations such as geometric transformation processing, color enhancement processing, and advanced enhancement processing on the initial image data, and then obtain an adversarial generative network model that can use an original input image to generate image data of various styles through training of the initial image data set.

[0102] Optionally, in some feasible embodiments of the method for generating multi-view image data of the present application, in addition to invoking the adversarial generative network model to expand the initial view image data to obtain multi-view image data and train the adversarial generative network model, the terminal device can also train a target detection network model based on the obtained multi-view image data. That is, after the above-mentioned operation of "obtaining multi-view image data corresponding to the initial view image data", the method for generating multi-view image data of the present application can also include:

[0103] Annotating the multi-view image data to obtain sample data annotation;

[0104] A target detection network model is obtained based on the sample data annotation and the multi-view image data training, so as to process the real-time environment image data through the target detection network model to obtain a model prediction result.

[0105] In this embodiment, when expanded multi-view image data is obtained through an adversarial generative network, the model training unit annotates the multi-view image to obtain sample data annotations, and inputs the sample data annotations and the multi-view image data into a preset initial detection network model, so as to train the initial detection network model through the sample data annotations and the multi-view image data to obtain a target detection network model, and then process the acquired real-time environmental image data based on the target detection network to obtain a model prediction result.

[0106] In this way, the model training unit can obtain sample data annotation by annotating the obtained multi-view image data, thereby ensuring that the image data contains information such as target coordinates and target categories, and then a more accurate target detection network model can be obtained through sample data annotation and multi-view image data training.

[0107] Optionally, in some feasible embodiments of the method for generating multi-view image data of the present application, the operation of processing the real-time environment image data by the target detection network model to obtain the model prediction result may include:

[0108] Acquire real-time environment image data, and process the real-time environment image data through the target detection network model to obtain image feature information corresponding to the real-time environment image data;

[0109] A model prediction result is obtained based on the image feature information.

[0110] In this embodiment, after the target detection network model is trained, when the terminal device detects that the vehicle is in a driving state, it can call the image acquisition device configured in the vehicle to shoot the environment around the vehicle to obtain real-time environment image data corresponding to the surrounding environment. The terminal device can input the real-time environment image data into the trained target detection network model, and the target detection network extracts the image feature information contained in the real-time environment image, and obtains the prediction results of various models such as the passable area contained in the real-time environment image based on the image feature information.

[0111] Exemplarily, for example, after the target detection network model is trained, the terminal device deploys the target detection network model in the control system in the vehicle. When the vehicle is driving, the vehicle control system can call the fisheye cameras configured around the vehicle to shoot the surrounding environment to obtain various real-time environment image data, and input each of the real-time environment image data into the target detection network model. The target detection network model processes each real-time environment image data to extract the image feature information contained in each real-time environment image data, and determines the passable area and background area contained in the real-time environment image based on each of the image feature information, thereby determining the passable area and the background area as the model prediction results, and the vehicle control system then plans the route based on the model prediction results.

[0112] In this way, when the vehicle is running, the terminal device can input real-time environmental image data containing the vehicle's surrounding environment into the trained target detection network model, so that the target detection network model obtains a model prediction result based on the real-time environmental image data, and plans the route based on the model prediction result, thereby improving the user's driving experience.

[0113] Furthermore, based on the first embodiment of the method for generating multi-view image data of the present application, a second embodiment of the method for generating multi-view image data of the present application is proposed.

[0114] In the second embodiment of the method for generating multi-view image data of the present application, after the above operation of "obtaining a target detection network model based on the sample data annotation and the multi-view image data training", the following may also be included:

[0115] Get the preset validation data set;

[0116] The target detection network model is verified based on the verification data set to obtain an average model accuracy corresponding to the target detection network model.

[0117] In this embodiment, after training the target detection network model, the terminal device can also obtain a verification data set consisting of image data taken from different vehicles at different perspectives, and input the verification data set into the target detection network, thereby determining the average model accuracy corresponding to the target detection network based on the sample category corresponding to the image data output by the target detection network.

[0118] Exemplarily, for example, before training the target detection network model, the terminal device first obtains a data set of 256×256 in size, including 50,000 images of 776 vehicles from different perspectives, and divides each data set into a training set, a validation set, and a test set in a ratio of 5:2:3. After that, the terminal obtains the target detection network model through training with the training set, and inputs the image data contained in the validation set into the trained target detection network model to obtain the image data output by the target detection network model, and determines the number of samples corresponding to each of the sample types such as true positive samples TP, false positive samples FP, true negative samples TN, and false negative samples FN corresponding to the output image data, and calculates the accuracy based on the preset formula:

[0119]

[0120] The model accuracy corresponding to the target detection network model is calculated.

[0121] It should be noted that, in the present embodiment, in the above accuracy calculation formula, P=TP / (TP+FP), R=TP / (TP+FN), and C is the number of categories.

[0122] In this way, the terminal device can determine the model accuracy result corresponding to the target detection network model accurately calculated based on the verification data set, and then judge whether the training of the target detection network model is successful based on the model accuracy result.

[0123] In addition, in this embodiment and another embodiment, the terminal device may also determine the accuracy of the target detection network model based on the IoU threshold, that is, the terminal device may determine the overlap ratio between the target image output by the target detection network model and the initial image border:

[0124] IoU=S overlap / S union ;

[0125] Thus, the model accuracy corresponding to the target detection network is calculated. It can be understood that, in this embodiment, the above S overlap represents the intersection area of ​​the predicted image and the real image bounding box, S union Identify the union between the predicted image and the true image.

[0126] In addition, please refer to Figure 6 , Figure 6 This is a diagram of the experimental results of the traditional detection network model involved in the first embodiment of the method for generating multi-view image data provided by the embodiment of the present application, such as Figure 6 As shown in FIG. 1 , the public VOC2007 complete training set, VOC2007-10% training set, and VOC, VOC2007-10% data augmented training set were input into the traditional detection network model used in the related art and trained for 50 epochs. After the training was completed, the above accuracy test was performed on the trained traditional detection network model. Based on the following Figure 6 From the test results shown, it can be seen that when the number of training samples is reduced to 10%, the corresponding AP value of the model decreases from 0.95 to 0.68. At this time, the model cannot learn all the features of the vehicle target, resulting in a decrease in recognition accuracy. Similarly, when the number of training samples is reduced to 10%, even with the use of data enhancement training sets, the model still cannot learn all the features of the vehicle target, so the AP value does not change significantly.

[0127] Please refer to Figure 7 , Figure 7 The experimental result diagram of the target detection network involved in the first embodiment of the method for generating multi-view image data provided in the embodiment of the present application is shown. The VOC2007-10% training set is input into the adversarial network model used in the embodiment of the present application for sample expansion, and the above target detection network model is obtained by training the training set after sample expansion, and the network is trained with vehicle pictures from two angles in the training set to obtain experimental result A, the vehicle pictures from a single angle are trained to obtain experimental result B, the front view of the vehicle in the training set and the side view of the vehicle generated by the adversarial network are trained to obtain experimental result C, and the side view of the vehicle in the training set and the front view of the vehicle generated by the adversarial network are trained to obtain experimental result D. Figure 7 As shown, compared with experimental result A and experimental result B, the accuracy of the model decreased by 21.58%, indicating that the target detection network model cannot be trained based on vehicle images of a single perspective. Based on experimental results B and experimental results C, it can be confirmed that the target detection network trained in the embodiment of the present application can understand more features of the vehicle, and the model accuracy is improved by 8.43%. Based on experimental results C and experimental results D, it can be confirmed that the target detection network trained in the embodiment of the present application is stable for image generation of two-way views.

[0128] Furthermore, based on the first embodiment and / or the second embodiment of the method for generating multi-view image data of the present application, a preferred embodiment of the method for generating multi-view image data of the present application is proposed.

[0129] Please refer to Figure 3 and Figure 4 ,in, Figure 3 This is a schematic diagram of the steps of a preferred embodiment of a method for generating multi-view image data provided in an embodiment of the present application. Figure 4 This is a schematic diagram of the image processing flow involved in an embodiment of a method for generating multi-view image data provided in an embodiment of the present application, such as Figure 3 As shown, in this embodiment, the terminal device first collects initial image data corresponding to multiple targets (ie Figure 3 The initial image data is enhanced to obtain enhanced image data, and then the initial image data and the enhanced image data are integrated to generate the initial data set (i.e. Figure 3 The terminal device inputs the generated initial data set into the model training unit, and the model training unit trains the adversarial generative network (i.e., Figure 3 After that, the terminal device obtains an initial view image dataset (i.e., a target detection dataset) for training the target detection network, and inputs the initial view image dataset into the model training unit, which then inputs the initial view image dataset into the adversarial generation network model. Figure 4 As shown, the adversarial generative network inputs the acquired initial perspective image data into the internally configured mapping network layer respectively, and inputs the preset style control vector into the generator through the mapping network layer. At the same time, the adversarial generative model inputs the initial perspective image data into the semantic segmentation network, and cuts the initial perspective image data through the semantic segmentation network to obtain the target area and background area corresponding to the initial perspective image, and generates mask vectors corresponding to the target area and the background area respectively. The semantic segmentation network then inputs the mask vector into the generator, and the generator processes the initial perspective image data based on the acquired style control vector and mask vector to generate corresponding virtual images and real images. The adversarial generative network model calls the discriminator to determine the authenticity of the image output by the generator. The adversarial generative network model updates the generator based on the authenticity and calculates the overall loss value in the update process, and determines that the update is completed when the overall loss value is less than the preset loss value threshold to obtain the target generator. The adversarial generative network processes the initial perspective image data through the target generator to obtain the corresponding multi-perspective image data. The adversarial generative network model also annotates the generated multi-perspective image data with sample data. Finally, the model training unit obtains the target detection network model based on the sample data annotation and the multi-perspective image data training, and deploys the target detection network model in the vehicle's control system.

[0130] In addition, an embodiment of the present application further provides a device for generating multi-view image data. The device for generating multi-view image data of the present application is also applied to the terminal device as described above.

[0131] Please refer to Figure 8 , the device for generating multi-view image data of the present application may include:

[0132] An image acquisition module, configured to acquire preset initial viewing angle image data;

[0133] The image processing module is configured to perform multi-style processing on the initial perspective image data through a preset adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data.

[0134] In addition, an embodiment of the present application also provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for generating multi-view image data as described above.

[0135] In addition, an embodiment of the present application further provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for generating multi-view image data as described above are implemented.

[0136] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0138] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating multi-view image data, characterized in that: The method comprises: Acquire initial viewing angle image data; The initial perspective image data is subjected to multi-style processing through an adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data.

2. The method for generating multi-view image data according to claim 1, characterized in that: The initial perspective image data is subjected to multi-style processing by using an adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data, including: Segmenting the initial perspective image data through an adversarial generative network model to obtain a target area and a background area corresponding to the initial perspective image data; Generate mask vectors corresponding to the target area and the background area respectively; Multi-style processing is performed on the initial view image data based on the mask vector and the style control vector to obtain multi-view image data.

3. The method for generating multi-view image data according to claim 2, characterized in that: The method further comprises: performing multi-style processing on the initial view image data based on the mask vector and the style control vector to obtain multi-view image data, including: Performing multi-style processing on the initial perspective image data based on the mask vector and the style control vector to generate corresponding virtual images and real images; determining authenticity between the virtual image and the real image; Based on the authenticity, a generator in the adversarial generative network model is updated to obtain a target generator; The initial view image data is processed based on the target generator to obtain multi-view image data.

4. The method for generating multi-view image data according to claim 1, wherein: Before acquiring the initial viewing angle image data, the method further includes: Acquire a plurality of initial image data, and process the plurality of initial image data respectively to generate a plurality of enhanced image data; Integrate the target image data in the plurality of the initial image data and the target enhanced data corresponding to the target image data in the plurality of the enhanced image data to obtain an initial image data set; The adversarial generative network model is obtained by training based on the initial image data set.

5. The method for generating multi-view image data according to claim 1, wherein: After obtaining the multi-view image data corresponding to the initial view image data, the method further includes: Annotating the multi-view image data to obtain sample data annotation; A target detection network model is obtained based on the sample data annotation and the multi-view image data training, so as to process the real-time environment image data through the target detection network model to obtain a model prediction result.

6. The method for generating multi-view image data according to claim 5, characterized in that: The real-time environment image data is processed by the target detection network model to obtain a model prediction result, including: Acquire real-time environment image data, and process the real-time environment image data through the target detection network model to obtain image feature information corresponding to the real-time environment image data; A model prediction result is obtained based on the image feature information.

7. The method for generating multi-view image data according to claim 5, characterized in that: After obtaining the target detection network model based on the sample data annotation and the multi-view image data training, the method further includes: Get the preset validation data set; The target detection network model is verified based on the verification data set to obtain an average model accuracy corresponding to the target detection network model.

8. A device for generating multi-view image data, characterized in that: The device comprises: An image acquisition module, configured to acquire initial viewing angle image data; The image processing module is configured to perform multi-style processing on the initial perspective image data through an adversarial generative network model to obtain multi-perspective image data corresponding to the initial perspective image data.

9. A terminal device, characterized in that: The terminal device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for generating multi-view image data according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for generating multi-view image data according to any one of claims 1 to 7 are implemented.

Images