Three-dimensional model texture mapping method, system and equipment based on content awareness

Abstract

The invention discloses a three-dimensional model texture mapping method, system and equipment based on content awareness, and the method comprises the steps: carrying out the mask segmentation of image data of a target region through a mask region convolutional neural network model, and obtaining a moving target mask of the image data of a moving target in the target region; a moving target in the image data is effectively sensed and extracted; triangular mesh region division is performed on a target region so as to effectively enhance the pertinence of data processing on the target region, and the image data of all the triangular mesh regions containing the moving target mask are counted, so that the triangular mesh regions with the moving target mask are analyzed in a targeted manner, and then mask image data are obtained through carding; by comparing the moving target mask with the mask image data, the target mask of the moving target which does not exist in the mask image data is determined in a targeted manner, and finally the target three-dimensional texture model of the target area is determined according to the target mask, so that the moving target is effectively removed.

Description

Technical Field

[0001] The present invention relates to the field of surveying and mapping technology, and in particular to a three-dimensional model texture mapping method, system and device based on content perception. Background Art

[0002] As the concept of new basic surveying and mapping gradually becomes popular and the technology and standard system of real-scene 3D China continues to improve, the construction paths at all levels are gradually becoming clear, and real-scene 3D as a new basic surveying and mapping standardized product has gradually become the focus of the industry. In the texture mapping process of 3D reconstruction, the wrong projection problem caused by moving targets interferes with the selection of correct texture information, resulting in texture distortion in the generated 3D model scene due to incorrect texture selection. In the actual application process (especially in the fields of smart transportation and unmanned driving), real and accurate 3D model data is usually required.

[0003] At present, the existing method of obtaining 3D model data is mainly to combine and fill the images of the area itself. However, this method cannot effectively solve the reconstruction problem caused by the movement of objects such as vehicles and shadows. In addition, although there are also image editing software for adaptive editing, it simply erases all the target information in the image, which may cause distortion in the subsequently reconstructed 3D scene.

[0004] Therefore, in the process of constructing a three-dimensional model in the prior art, there is a problem that the influence of moving targets cannot be effectively removed. Summary of the invention

[0005] In view of this, it is necessary to provide a content-aware three-dimensional model texture mapping method, system and device to solve the problem that the existing technology cannot effectively remove the influence of moving targets in the process of building a three-dimensional model.

[0006] In order to solve the above problems, the present invention provides a 3D model texture mapping method based on content perception, comprising:

[0007] Obtain image data of the target area, and perform mask segmentation on the image data according to a well-trained target mask area convolutional neural network model to obtain a moving target mask of the image data;

[0008] Dividing the target area into triangular mesh areas, matching the triangular mesh areas with the image data, and then counting the triangular mesh areas where the moving target mask exists to obtain mask image data;

[0009] Determining the target mask in the mask image data by comparing the moving target mask with the mask image data;

[0010] A target 3D texture model of the target area is determined according to the target mask.

[0011] Furthermore, the target mask region convolutional neural network model includes a feature extraction network, a region proposal network, a region of interest matching layer, a fully connected layer, a fully convolutional network and a lightweight network; the image data is masked and segmented according to the well-trained target mask region convolutional neural network model to obtain a moving target mask of the image data, including:

[0012] The feature extraction network extracts the features of the image data, the region proposal network divides the image data into regions based on the features, and then the region of interest matching layer determines the region of interest of the image data based on the region division results;

[0013] The fully connected layer determines the category and location of the region of interest, and the lightweight network determines the color category of the region of interest;

[0014] The fully convolutional network performs mask operations on the region of interest to obtain a moving target mask for the image data;

[0015] The moving object mask includes the category, position and color category of the region of interest.

[0016] Further, the target area is divided into triangular mesh areas, and the triangular mesh areas are matched with the image data, and then the triangular mesh areas where the moving target mask exists are counted to obtain the mask image data, including:

[0017] Divide the target area into triangular mesh areas to obtain corresponding triangular mesh areas;

[0018] Traversing the mask data of the image data, and determining all the image data including the same triangular mesh area as the image data group of the triangular mesh area;

[0019] When any image data in the image data group contains a moving target mask, the image data group is mask image data;

[0020] When the image data set does not include a moving target mask, the image data set is fixed mask image data.

[0021] Furthermore, when any image data in the image data group contains a moving target mask, the image data group is mask image data, and further includes:

[0022] Counting the number of moving target masks and the number of mask image data respectively, and calculating the ratio of the number of moving target masks to the number of mask image data as a first ratio;

[0023] When the first ratio is less than a first preset threshold, it is determined that a small number of moving targets exist in the triangular mesh area corresponding to the moving target mask;

[0024] When the first ratio is less than the second preset threshold, it is determined that a large number of moving objects exist in the triangular mesh area corresponding to the moving object mask.

[0025] Furthermore, when any image data in the image data group contains a moving target mask, the image data group is mask image data, and further includes:

[0026] Count the color categories of the moving target mask and construct a moving target mask color histogram;

[0027] Obtaining a moving target mask maximum value of a moving target mask color histogram, and calculating a ratio of the moving target mask maximum value to the number of moving target masks as a second ratio;

[0028] When the second ratio is less than the third preset threshold, it is determined that a changing moving object exists in the triangular mesh area corresponding to the moving object mask.

[0029] Further, determining the target mask in the mask image data according to the moving target mask and the mask image data includes:

[0030] Count the number of moving target masks and the number of mask image data respectively;

[0031] When the number of moving target masks is less than the number of mask image data, determining the mask data corresponding to the image data not including the moving target mask as the target mask;

[0032] When the number of moving target masks is equal to the number of mask image data, it is determined that there is a fixed moving target on the triangular mesh area corresponding to the moving target mask, and any mask data is determined to be a target mask.

[0033] Further, determining a target three-dimensional texture model of the target area according to the target mask includes:

[0034] Obtaining an initial 3D texture model of the target area, and filling the target mask into the initial 3D texture model to obtain a modified 3D texture model;

[0035] The misaligned textures in the modified 3D texture model are eliminated according to the random sample consensus algorithm, and the missing areas are filled to obtain the target 3D texture model.

[0036] In order to solve the above problems, the present invention further provides a content-aware 3D model texture mapping system, comprising:

[0037] A moving target mask acquisition module is used to acquire image data of a target area and perform mask segmentation on the image data according to a well-trained target mask area convolutional neural network model to obtain a moving target mask of the image data;

[0038] A mask image data acquisition module, used to divide the target area into triangular mesh areas, match the triangular mesh areas with the image data, and then count the triangular mesh areas where the moving target mask exists to obtain mask image data;

[0039] A target mask determination module, used for comparing the moving target mask with the mask image data to determine the target mask in the mask image data;

[0040] The target 3D texture model module is used to determine the target 3D texture model of the target area according to the target mask.

[0041] In order to solve the above problems, the present invention also provides a content-aware three-dimensional model texture mapping device, including a processor and a memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the content-aware three-dimensional model texture mapping method described above is implemented.

[0042] In order to solve the above problem, the present invention also provides a storage medium, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the content-aware three-dimensional model texture mapping method as described above.

[0043] The beneficial effects of adopting the above embodiments are as follows: the present invention provides a content-aware three-dimensional model texture mapping method, system and device, which performs mask segmentation on the image data of the target area by training a complete target mask area convolutional neural network model, so as to obtain the moving target mask of the image data of the moving target in the target area in a targeted manner, and effectively perceive and extract the moving target in the image data; by dividing the target area into triangular mesh areas, the targeted data processing of the target area is effectively enhanced, and by targeted statistics of the image data of all triangular mesh areas containing moving target masks, the triangular mesh areas with moving target masks are analyzed in a targeted manner, and then the mask image data is sorted out; by comparing the moving target mask with the mask image data, the target mask of the mobile target not in the mask image data is determined in a targeted manner, and finally the target three-dimensional texture model of the target area is determined according to the target mask, and the mobile target is effectively removed. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a flow chart of an embodiment of a content-aware 3D model texture mapping method provided by the present invention;

[0045] Figure 2 A schematic diagram of the structure of an embodiment of a target mask region convolutional neural network model provided by the present invention;

[0046] Figure 3 A schematic diagram of a flow chart of an embodiment of a moving target mask for obtaining image data provided by the present invention;

[0047] Figure 4 A schematic diagram of a process of obtaining mask image data according to an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of a flow chart of an embodiment of determining a target mask in mask image data provided by the present invention;

[0049] Figure 6 A schematic diagram of a flow chart of an embodiment of determining a target three-dimensional texture model of a target area provided by the present invention;

[0050] Figure 7 A structural block diagram of an embodiment of a content-aware 3D model texture mapping system provided by the present invention;

[0051] Figure 8 This is a structural block diagram of an embodiment of a content-aware three-dimensional model texture mapping device provided by the present invention. DETAILED DESCRIPTION

[0052] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0053] As the concept of new basic surveying and mapping gradually becomes popular and the technology and standard system of real-scene 3D China continues to improve, the construction paths at all levels are gradually becoming clear, and real-scene 3D as a new basic surveying and mapping standardized product has gradually become the focus of the industry. In the texture mapping process of 3D reconstruction, the wrong projection problem caused by moving targets interferes with the selection of correct texture information, resulting in texture distortion in the generated 3D model scene due to incorrect texture selection. In the actual application process (especially in the fields of smart transportation and unmanned driving), real and accurate 3D model data is usually required.

[0054] At present, the existing method of obtaining 3D model data is mainly to combine and fill the images of the area itself. However, this method cannot effectively solve the reconstruction problem caused by the movement of objects such as vehicles and shadows. In addition, although there are also image editing software for adaptive editing, it simply erases all the target information in the image, which may cause distortion in the subsequently reconstructed 3D scene.

[0055] Therefore, in the process of constructing a three-dimensional model in the prior art, there is a problem that the influence of moving targets cannot be effectively removed.

[0056] In order to solve the above problems, the present invention provides a content-aware three-dimensional model texture mapping method, system and device, which are described in detail below.

[0057] Figure 1 A flow chart of an embodiment of a content-aware 3D model texture mapping method provided by the present invention is shown in FIG. Figure 1 As shown, the content-aware 3D model texture mapping method includes:

[0058] Step S101: acquiring image data of a target area, and performing mask segmentation on the image data according to a well-trained target mask area convolutional neural network model to obtain a moving target mask of the image data;

[0059] Step S102: dividing the target area into triangular mesh areas, matching the triangular mesh areas with the image data, and then counting the triangular mesh areas where the moving target mask exists to obtain mask image data;

[0060] Step S103: determining the target mask in the mask image data by comparing the moving target mask with the mask image data;

[0061] Step S104: determining a target 3D texture model of the target area according to the target mask.

[0062] In this embodiment, the image data of the target area is masked and segmented by training a complete target mask area convolutional neural network model, so as to obtain the moving target mask of the image data of the moving target in the target area in a targeted manner, and effectively perceive and extract the moving target in the image data; the target area is divided into triangular mesh areas to effectively enhance the targeted data processing of the target area, and the image data of all triangular mesh areas containing moving target masks are targetedly counted to achieve targeted analysis of the triangular mesh areas with moving target masks, and then the mask image data is sorted out; by comparing the moving target mask with the mask image data, the target mask of the mobile target that is not in the mask image data is targeted, and finally the target three-dimensional texture model of the target area is determined according to the target mask, and the mobile target is effectively removed.

[0063] Obviously, in this embodiment, by performing moving target identification on the image data of the target area and obtaining a moving target mask, and by performing targeted statistics on the image data of all triangular mesh areas containing the moving target mask, it is achieved that for any triangular mesh area with a moving target, there is at least one mask image data corresponding to it, and the target mask corresponding to the triangular mesh area can be determined, so as to finally determine the target three-dimensional texture model of the target area.

[0064] It should be noted that the mask image data not only includes the result of the triangular mesh area division, but also marks the corresponding moving target information.

[0065] As a preferred embodiment, in step S101, the target mask region convolutional neural network model includes a feature extraction network, a region proposal network, a region of interest matching layer, a fully connected layer, a fully convolutional network and a lightweight network, such as Figure 2 As shown, Figure 2 A schematic structural diagram of an embodiment of a target mask area convolutional neural network model provided by the present invention.

[0066] Furthermore, after acquiring the image data of the target area, in order to obtain a moving target mask of the image data, such as Figure 3 As shown, Figure 3 A schematic diagram of a flow chart of an embodiment of obtaining a moving target mask of image data provided by the present invention includes:

[0067] Step S111: the feature extraction network extracts features of the image data, the region proposal network divides the image data into regions according to the features, and then the region of interest matching layer determines the region of interest of the image data according to the region division result;

[0068] Step S112: The fully connected layer determines the category and position of the region of interest, and the lightweight network determines the color category of the region of interest;

[0069] Step S113: the full convolutional network performs a mask operation on the region of interest to obtain a moving target mask of the image data;

[0070] The moving object mask includes the category, position and color category of the region of interest.

[0071] In this embodiment, by performing feature extraction and region allocation on the image data, targeted regional division of the image data is achieved. On the basis of determining the category and position of the region of interest, the lightweight network further classifies the region of interest into color categories to determine the color category of the region of interest. Finally, the fully convolutional network performs a mask operation on the region of interest to mark the category, position, and color category of the region of interest in the image data one by one, thereby obtaining a moving target mask of the image data, which can fully and meticulously display the area information in the image data where there is a moving target and needs to be focused on for replacement.

[0072] It should be noted that the moving object mask in the present application includes three features: category, position and color category of the region of interest.

[0073] As a preferred embodiment, in step S102, since the amount of image data is relatively large, in order to obtain the required mask image data in a targeted manner, such as Figure 4 As shown, Figure 4 A schematic diagram of a process of obtaining mask image data according to an embodiment of the present invention includes:

[0074] Step S121: Divide the target area into a triangular mesh area to obtain a corresponding triangular mesh area;

[0075] Step S122: traversing the mask data of the image data, and determining that all image data including the same triangular mesh area is an image data group of the triangular mesh area;

[0076] Step S123: when any image data in the image data set contains a moving target mask, the image data set is mask image data;

[0077] Step S124: When the image data set does not include a moving target mask, the image data set is fixed mask image data.

[0078] In this embodiment, the target area is first divided into triangular mesh areas to improve the specificity of subsequent data processing and obtain the corresponding triangular mesh areas; then, all image data are sorted out using the divided triangular mesh areas as classification standards, and all image data including the same triangular mesh area are determined to be a group, thereby obtaining the corresponding image data group.

[0079] It should be noted that the process of dividing the image data corresponding to the triangular mesh area does not involve the judgment of the moving target. Therefore, it is necessary to further screen the image data group to reduce unnecessary subsequent data sorting work.

[0080] Specifically, when any image data in the image data set contains a moving target mask, the image data set is mask image data; when the image data set does not contain a moving target mask, the image data set is fixed mask image data. In other words, all image data sets need to be further screened to ensure that all mask image data contain a moving target mask corresponding to the moving target, to ensure the necessity of subsequent data processing.

[0081] Furthermore, after determining the mask image data, in order to ensure the reliability of the mask image data, it is also necessary to determine the state of the corresponding moving target. Specifically, the number of moving target masks and the number of mask image data are counted respectively, and the ratio of the number of moving target masks to the number of mask image data is calculated as the first ratio; when the first ratio is less than the first preset threshold, it is determined that a small number of moving targets exist in the triangular mesh area corresponding to the moving target mask; when the first ratio is less than the second preset threshold, it is determined that a large number of moving targets exist in the triangular mesh area corresponding to the moving target mask.

[0082] In this embodiment, by using the first ratio as the basis for judgment, adaptive judgment is made on the state of the moving target in the mask image data, which can better avoid data errors caused by accidental errors in the process of mobile target judgment by the target mask area convolutional neural network model. By making a detailed division of the state of the moving target, the convenience of subsequent data review can be improved.

[0083] In a specific embodiment, the value range of the first preset threshold is set to 0 to 0.08, preferably 0.05; the value range of the second preset threshold is set to 0.8 to 0.95, preferably 0.9.

[0084] In other embodiments, the values ​​of the first preset threshold and the second preset threshold may be adjusted according to actual conditions, which will not be elaborated herein.

[0085] Furthermore, in order to effectively utilize the color category characteristics of the moving target mask, in order to determine whether the moving target has changed in the image data corresponding to the mask image data, that is, since the process of acquiring image data has a certain randomness, the time interval between different image data may be relatively long. Therefore, in order to determine whether the moving target has changed, first, the color category of the moving target mask is counted, and a moving target mask color histogram is constructed; then, the moving target mask maximum value of the moving target mask color histogram is obtained, and the ratio of the moving target mask maximum value to the number of moving target masks is calculated as the second ratio; when the second ratio is less than the third preset threshold, it is determined that there is a changed moving target in the triangular mesh area corresponding to the moving target mask.

[0086] In this embodiment, by taking the color category of the moving target mask as the basis for judgment and calculating the second ratio in combination with the maximum value of the moving target mask, while excluding the recognition error of the convolutional neural network model in the target mask area, it is further determined whether the moving target in the triangular mesh area corresponding to the moving target mask has changed. This not only reversely verifies the recognition effect of the convolutional neural network model in the target mask area, but also facilitates the adaptive revision of the triangular mesh area during subsequent data processing.

[0087] In a specific embodiment, the value range of the third preset threshold is set to 0.75 to 0.9, preferably 0.8.

[0088] In other embodiments, the value of the third preset threshold may be adjusted according to actual conditions, which will not be elaborated here.

[0089] In a specific embodiment, the state of the moving object is determined by utilizing the consistency characteristics of multiple view images in the same area.

[0090] Specifically, it is assumed that the geometric structure of the scene generated at this time is represented by a triangular mesh as S = {f i}(i=1,2,...,n), the acquired image data set is represented as I={I i}(i=1,2,...,m), the identified target mask set is represented as Mask(I i )={mask j}(j=1,2,...,n i ), a single image I calculated by the SFM algorithm i The perspective projection matrix is ​​Π i :R 3 →R 2 Then, using Π i The triangle face f in the triangle mesh S i Projected to image I i Combined with depth detection, image I can be obtained i The visible triangle set S i =Π i (S). The specific determination steps are as follows:

[0091] Traverse each image I i , sequentially take out its target mask set Mask(I i ) in the unprocessed mask j According to the image I i The visible triangle set S i , count the projection area in the mask j The triangles within the range form a local area patch j .

[0092] Count other patches that have not been traversed and are in the area j The number of equally visible images n k , get this n k The number n1 of target masks marked at the corresponding projection positions on the image. If the first ratio ratio1 = n1 / n k <δ0 (0<δ0<0.08), it means that only a very small number of targets pass through the patch during image acquisition. j , then we can determine the mask j The targets corresponding to the n1 masks with the same name are in a moving state. The judgment results are recorded and the next mask that has not been judged is checked in sequence. j+1 Otherwise, proceed to the next step.

[0093] If the first ratio ratio1<δ1(0.8<δ1<0.95), it means that there is a period of time during the image acquisition process. j If there is a target on the mask, then the judgment result of the n1+1 target mask is recorded as a stationary target, and the next mask that has not been judged is recorded in sequence. j+1 Otherwise, it means the local area patch j If there is a stationary target, proceed to the next step.

[0094] Count the histograms of the colors corresponding to the n1 masks and get the maximum value n2 among them. If the second ratio ratio2=n2 / n1>δ2(0.75<δ2<0.9), it means that patch j If the color information of the above targets is consistent, they should be the same target, then it can be determined that the n1+1 target masks are stationary targets. Otherwise, it means that in patch j The target that has been stopped has changed, so it can be determined that these n1+1 target masks are stationary targets, record the current judgment result and process the next mask that has not been judged in sequence.

[0095] As a preferred embodiment, in step S103, after determining the moving target mask and the mask image data, it is also necessary to determine the target mask in the mask image data as a basis for subsequent data filling, such as Figure 5 As shown, Figure 5 A schematic diagram of a flow chart of an embodiment of determining a target mask in mask image data provided by the present invention includes:

[0096] Step S131: Counting the number of moving target masks and the number of mask image data respectively;

[0097] Step S132: when the number of moving target masks is less than the number of mask image data, determining the mask data corresponding to the image data not including the moving target mask as the target mask;

[0098] Step S133: When the number of moving target masks is equal to the number of mask image data, it is determined that there is a fixed moving target on the triangular mesh area corresponding to the moving target mask, and any mask data is determined to be a target mask.

[0099] In this embodiment, first, the number of moving target masks and the number of mask image data are counted respectively; then, the number of the two is used as a criterion to determine the origin of the target mask. Specifically, when the number of moving target masks is less than the number of mask image data, it means that there is a part in the mask image data that does not contain the moving target. Therefore, the mask data corresponding to the image data that does not contain the moving target mask is directly determined to be the target mask; and when the number of moving target masks is equal to the number of mask image data, it is determined that there is a fixed moving target on the triangular mesh area corresponding to the moving target mask, that is, the moving target identified by the convolutional neural network model of the target mask area has been staying on the triangular mesh area, which is a part that cannot be processed, and therefore, any mask data is determined to be the target mask, that is, the moving target on the triangular mesh area cannot be adaptively removed.

[0100] As a preferred embodiment, in step S104, after determining the target mask, it is also necessary to determine the target 3D texture model of the target area, such as Figure 6 As shown, Figure 6 A schematic diagram of a process for determining a target three-dimensional texture model of a target area provided by the present invention includes:

[0101] Step S141: obtaining an initial 3D texture model of the target area, and filling the target mask into the initial 3D texture model to obtain a modified 3D texture model;

[0102] Step S142: removing the misaligned textures in the modified three-dimensional texture model according to the random sample consistency algorithm, and filling the missing areas to obtain the target three-dimensional texture model.

[0103] In this embodiment, first, an initial three-dimensional texture model of the target area is obtained, and the target mask is filled into the initial three-dimensional texture model to obtain a corrected three-dimensional texture model; then, the misaligned texture in the corrected three-dimensional texture model is eliminated according to the random sample consistency algorithm, and the missing area is filled to obtain the target three-dimensional texture model.

[0104] In this embodiment, the initial three-dimensional texture model of the target area is filled accordingly through the target mask, so that the moving targets existing in the initial three-dimensional texture model can be adaptively removed, and the cleanliness of the revised three-dimensional texture model is improved, so as to obtain the most accurate information of the target area; further, in order to avoid errors caused by improper data processing, it is also necessary to eliminate the misaligned textures in the corrected three-dimensional texture model according to the random sample consistency algorithm, and fill the missing areas, that is, to check whether the revised three-dimensional texture model has errors, so as to obtain the final target three-dimensional texture model.

[0105] Through the above method, the image data of the target area is masked and segmented by training a complete target mask area convolutional neural network model, so as to obtain the moving target mask of the image data of the moving target in the target area in a targeted manner, and effectively perceive and extract the moving target in the image data; by dividing the target area into triangular mesh areas, the targeted data processing of the target area is effectively enhanced, and by targeted statistics of all image data of the triangular mesh areas containing the moving target mask, the triangular mesh areas where the moving target mask exists are analyzed in a targeted manner, and then the mask image data is sorted out; by comparing the moving target mask with the mask image data, the target mask of the mobile target that is not in the mask image data is determined in a targeted manner, and finally the target three-dimensional texture model of the target area is determined according to the target mask.

[0106] In order to solve the above problems, the present invention also provides a 3D model texture mapping system based on content perception, such as Figure 7 As shown, Figure 7 This is a structural block diagram of an embodiment of a content-aware 3D model texture mapping system provided by the present invention. The content-aware 3D model texture mapping system 700 includes:

[0107] The moving target mask acquisition module 701 is used to acquire the image data of the target area, and perform mask segmentation on the image data according to the well-trained target mask area convolutional neural network model to obtain the moving target mask of the image data;

[0108] The mask image data acquisition module 702 is used to divide the target area into triangular mesh areas, match the triangular mesh areas with the image data, and then count the triangular mesh areas where the moving target mask exists to obtain mask image data;

[0109] A target mask determination module 703 is used to compare the moving target mask with the mask image data to determine the target mask in the mask image data;

[0110] The target 3D texture model module 704 is used to determine the target 3D texture model of the target area according to the target mask.

[0111] The present invention also provides a content-aware 3D model texture mapping device. Figure 8 As shown, Figure 8 The structural block diagram of an embodiment of a content-aware 3D model texture mapping device provided by the present invention. The content-aware 3D model texture mapping device 800 can be a computing device such as a mobile terminal, a desktop computer, a notebook, a PDA, and a server. The content-aware 3D model texture mapping device 800 includes a processor 801 and a memory 802, wherein the memory 802 stores a content-aware 3D model texture mapping program 803.

[0112] In some embodiments, the memory 802 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 802 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory 802 may also include both an internal storage unit of the computer device and an external storage device. The memory 802 is used to store application software and various types of data installed in the computer device, such as program codes for installing the computer device. The memory 802 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the content-aware three-dimensional model texture mapping program 803 may be executed by the processor 801, thereby realizing the content-aware three-dimensional model texture mapping method of each embodiment of the present invention.

[0113] In some embodiments, the processor 801 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 802, such as executing a content-aware three-dimensional model texture mapping program.

[0114] This embodiment also provides a computer-readable storage medium, on which a content-aware three-dimensional model texture mapping program is stored. When the computer program is executed by a processor, the content-aware three-dimensional model texture mapping method described above is implemented.

[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0116] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A content-aware 3D model texture mapping method, characterized in that: include: Acquire image data of the target area, and perform mask segmentation on the image data according to a well-trained target mask area convolutional neural network model to obtain a moving target mask of the image data; Dividing the target area into triangular mesh areas, matching the triangular mesh areas with the image data, and then counting the triangular mesh areas where the moving target mask exists to obtain mask image data; Determining the target mask in the mask image data by comparing the moving target mask with the mask image data; A target three-dimensional texture model of the target area is determined according to the target mask.

2. The content-aware 3D model texture mapping method according to claim 1, characterized in that: The target mask area convolutional neural network model includes a feature extraction network, a region proposal network, a region of interest matching layer, a fully connected layer, a fully convolutional network and a lightweight network; the image data is masked and segmented according to the well-trained target mask area convolutional neural network model to obtain a moving target mask of the image data, including: The feature extraction network extracts features of the image data, the region proposal network divides the image data into regions according to the features, and the region of interest matching layer determines the region of interest of the image data according to the region division result; The fully connected layer determines the category and position of the region of interest, and the lightweight network determines the color category of the region of interest; The fully convolutional network performs a mask operation on the region of interest to obtain a moving target mask of the image data; The moving object mask includes the category, position and color category of the region of interest.

3. The content-aware 3D model texture mapping method according to claim 2, characterized in that: The step of dividing the target area into triangular mesh areas, matching the triangular mesh areas with the image data, and then counting the triangular mesh areas where the moving target mask exists to obtain mask image data includes: Dividing the target area into triangular mesh areas to obtain corresponding triangular mesh areas; Traversing the mask data of the image data, and determining all the image data including the same triangular mesh area as an image data group of the triangular mesh area; When any image data in the image data group includes the moving object mask, the image data group is the mask image data; When the image data set does not include the moving target mask, the image data set is fixed mask image data.

4. The content-aware 3D model texture mapping method according to claim 3, characterized in that: When any image data in the image data group includes the moving target mask, the image data group is the mask image data, and further includes: Counting the number of moving target masks and the number of mask image data respectively, and calculating the ratio of the number of moving target masks to the number of mask image data as a first ratio; When the first ratio is less than a first preset threshold, it is determined that a small number of moving targets exist in the triangular mesh area corresponding to the moving target mask; When the first ratio is less than a second preset threshold, it is determined that a large number of moving objects exist in the triangular mesh area corresponding to the moving object mask.

5. The content-aware 3D model texture mapping method according to claim 3, characterized in that: When any image data in the image data group includes the moving target mask, the image data group is the mask image data, and further includes: Counting the color categories of the moving target mask and constructing a moving target mask color histogram; Acquire a moving target mask maximum value of the moving target mask color histogram, and calculate a ratio of the moving target mask maximum value to the number of moving target masks as a second ratio; When the second ratio is less than a third preset threshold, it is determined that a changing moving object exists in the triangular mesh area corresponding to the moving object mask.

6. The content-aware 3D model texture mapping method according to claim 3, characterized in that: The step of comparing the moving target mask with the mask image data to determine the target mask in the mask image data comprises: Count the number of moving target masks and the number of mask image data respectively; When the number of the moving target masks is less than the number of the mask image data, determining the mask data corresponding to the image data not including the moving target mask as the target mask; When the number of the moving target masks is equal to the number of the mask image data, it is determined that there is a fixed moving target on the triangular mesh area corresponding to the moving target mask, and any of the mask data is determined to be the target mask.

7. The content-aware 3D model texture mapping method according to claim 1, characterized in that: Determining the target three-dimensional texture model of the target area according to the target mask includes: acquiring an initial three-dimensional texture model of the target area, and filling the target mask into the initial three-dimensional texture model to obtain a modified three-dimensional texture model; The misaligned textures in the modified three-dimensional texture model are eliminated according to a random sample consensus algorithm, and the missing areas are filled to obtain a target three-dimensional texture model.

8. A content-aware 3D model texture mapping system, characterized in that: include: A moving target mask acquisition module is used to acquire image data of a target area, and perform mask segmentation on the image data according to a well-trained target mask area convolutional neural network model to obtain a moving target mask of the image data; A mask image data acquisition module, used to divide the target area into triangular mesh areas, match the triangular mesh areas with the image data, and then count the triangular mesh areas where the moving target mask exists to obtain mask image data; A target mask determination module, used for comparing the moving target mask with the mask image data to determine the target mask in the mask image data; The target three-dimensional texture model module is used to determine the target three-dimensional texture model of the target area according to the target mask.

9. A content-aware 3D model texture mapping device, characterized in that: It comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the content-aware three-dimensional model texture mapping method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores computer program instructions, and when the computer program instructions are executed by a computer, the computer is enabled to execute the content-aware three-dimensional model texture mapping method according to any one of claims 1 to 7.

Images