Method for generating tripartite graphs and neural network construction method for generating tripartite graphs
By performing image erosion and dilation using convolutional subnetworks connected in parallel within a neural network, the problem of slow generation speed of triangulation in existing technologies is solved, achieving efficient generation on GPUs or NPUs.
Patent Information
- Application Number
- CN202210075101.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-22
- Publication Date
- 2026-02-03
- Estimated Expiration
- 2042-01-22
AI Technical Summary
Existing methods for generating triangulation maps require manual annotation and are slow, especially when performing erosion and dilation operations on a CPU, which limits their widespread application.
Image erosion and dilation are performed using a pre-defined neural network, with GPU or NPU acceleration. The first and second convolutional sub-networks are connected in parallel to perform erosion and dilation respectively, and the output layer is used to generate a three-part image.
It significantly improves the speed of generating triangulation graphs, reduces costs and increases efficiency, and is suitable for inference processing on graphics processors and network processors.
Smart Images
Figure CN114298947B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method for generating tripartite images and a method for constructing a neural network for generating tripartite images. Background Technology
[0002] A trimap is a coarse division of a given image. For example... Figure 1 As shown, Figure 1 Given an image A, it is divided into foreground 13, background 11, and an unknown region 12, as shown below. Figure 1 As shown in Figure B.
[0003] In some solutions, trimas are manually generated by the user, requiring manual annotation of the given image, which is very user-unfriendly and impractical. In other solutions, erosion and dilation operations from the OpenCV software library are typically used to generate trimas for a given image. These erosion and dilation operations are performed on the Central Processing Unit (CPU), which is very slow, thus limiting the widespread adoption of trimas and related technologies.
[0004] The above background information is provided only to assist in understanding the inventive concept and technical solution of this application. It does not necessarily belong to the prior art of this application. In the absence of clear evidence that the above information was disclosed before the filing date of this application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention
[0005] In view of this, embodiments of this application provide a method for generating a tripartite graph and a neural network construction method for generating a tripartite graph, which can solve at least one technical problem in the related art.
[0006] In a first aspect, one embodiment of this application provides a method for generating a three-part image, comprising:
[0007] Obtain the image to be processed;
[0008] The image to be processed is input into a preset neural network to generate a triangulation image. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional subnetwork and the second convolutional subnetwork respectively erode and dilate the image to be processed. The output layer is used to generate the triangulation image based on the eroded image to be processed and the dilated image to be processed.
[0009] This embodiment uses a neural network to erode and dilate the image to be processed, and then generates a triangulation based on the eroded and dilated image. The neural network can infer on a graphics processing unit (GPU) or a neural-network processing unit (NPU), which greatly improves the speed of generating triangulation.
[0010] As one implementation of the first aspect, the step of inputting the image to be processed into a preset neural network to generate a triangulation image includes:
[0011] The image to be processed is input into the first convolutional subnetwork to obtain an erosion image; the image to be processed is input into the second convolutional subnetwork to obtain a dilation image;
[0012] The erosion image and the dilation image are added together to obtain the summed image;
[0013] The summed image is processed using a preset region limiting function to obtain a tripartite image, wherein the region limiting function has three sub-regions.
[0014] As one implementation of the first aspect, both the first convolutional subnetwork and the second convolutional subnetwork include a first convolutional layer, a second convolutional layer, and a rounding layer. The first convolutional layer uses a convolutional kernel to filter the image to be processed. The second convolutional layer adds a bias to the output of the first convolutional layer, and the bias is calculated based on the elements of the convolutional kernel. The rounding layer rounds down the output of the second convolutional layer.
[0015] As one implementation of the first aspect, the convolutional kernel is generated based on OpenCV's getStructuringElement function, and the calculation formula is as follows:
[0016]
[0017] Among them, T i For the i-th element of the convolution kernel, t i The i-th element of the initial generation kernel is generated using the getStructuringElement function, where i is an integer from 1 to j, and j is the total number of elements in the initial kernel.
[0018] As one implementation of the first aspect, for the first convolutional sub-network, the bias_ero is calculated according to the following formula:
[0019] bias_ero = -(A - thresh * B) + A,
[0020] Where A and B are constants, thresh = max{T1,T2,…,T i}
[0021] As one implementation of the first aspect, for the second convolutional sub-network, the bias_dil is calculated according to the following formula:
[0022] bias_dil = -(thresh*B) + A,
[0023] Where A and B are constants, thresh = max{T1,T2,…,T i}
[0024] As one implementation of the first aspect, the preset neural network further includes an input layer, which normalizes the image to be processed to obtain a normalized image, the first convolutional subnetwork and the second convolutional subnetwork respectively eroding and dilating the normalized image, and the output layer is used to generate the three-part image based on the eroded normalized image and the dilated normalized image.
[0025] As one implementation of the first aspect, the normalization of the image to be processed includes:
[0026] Determine the maximum and minimum values of each pixel in the image to be processed;
[0027] By using the maximum and minimum values, each pixel in the image to be processed is normalized to the target numerical range, resulting in a normalized image.
[0028] As one implementation of the first aspect, the input layer normalizes the image to be processed to the target numerical range [0,1].
[0029] As one implementation of the first aspect, before inputting the image to be processed into a preset neural network to generate a triangulation, the method further includes:
[0030] Determine the size and shape of the first and second convolution kernels;
[0031] The first convolutional kernel and the second convolutional kernel are generated based on the getStructuringElement function of OpenCV. The first bias is determined based on the first convolutional kernel, and the second bias is determined based on the second convolutional kernel.
[0032] The first convolution kernel and the first bias are assigned to the initial convolution kernel and the initial bias of the first convolutional sub-network in the initial neural network, and the second convolution kernel and the second bias are assigned to the initial convolution kernel and the initial bias of the second convolutional sub-network in the initial neural network to obtain the preset neural network.
[0033] As one implementation of the first aspect, the first convolutional subnetwork and the second convolutional subnetwork may share the first convolutional layer.
[0034] As one implementation of the first aspect, the preset neural network includes a plurality of first convolutional subnetworks connected in series, and / or the preset neural network includes a plurality of second convolutional subnetworks connected in series.
[0035] Secondly, one embodiment of this application provides a neural network construction method for generating triangulation maps, comprising:
[0036] Construct an initial neural network for generating the triangulation;
[0037] Determine the size and shape of the convolutional kernels of the initial neural network;
[0038] The getStructuringElement function based on OpenCV generates the elements of the convolution kernel, and the bias is determined based on the convolution kernel;
[0039] The elements and biases of the convolution kernel are assigned to the initial neural network to obtain the preset neural network;
[0040] The initial neural network has the same architecture as the preset neural network. The preset neural network is used to generate a triangulation of the image to be processed. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional subnetwork and the second convolutional subnetwork respectively erode and dilate the image to be processed. The output layer is used to generate the triangulation based on the eroded image to be processed and the dilated image to be processed.
[0041] This embodiment generates the initial neural network convolution kernel based on the getStructuringElement function, and then determines the bias based on the convolution kernel, thereby updating the initial neural network to obtain the preset neural network. This can greatly improve the construction efficiency of the neural network and greatly reduce the cost.
[0042] Thirdly, one embodiment of this application provides an apparatus for generating a three-part diagram, comprising:
[0043] The acquisition module is used to acquire the image to be processed;
[0044] A generation module is used to input the image to be processed into a preset neural network to generate a triangulation image; the preset neural network includes a backbone network and an output layer connected to the backbone network, the backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel, the first convolutional network and the second convolutional network respectively eroding and dilating the image to be processed, and the output layer is used to generate the triangulation image based on the eroded image to be processed and the dilated image to be processed.
[0045] Fourthly, one embodiment of this application provides a neural network construction apparatus for generating triangulations, comprising:
[0046] Build modules are used to construct the initial neural network for generating the triangulation.
[0047] The first determining module is used to determine the size and shape of the convolution kernel of the initial neural network;
[0048] The second determining module is used to generate the elements of the convolution kernel based on the getStructuringElement function of OpenCV, and determine the bias according to the convolution kernel;
[0049] The assignment module is used to assign the elements and biases of the convolution kernel to the initial neural network to obtain the preset neural network;
[0050] The initial neural network has the same architecture as the preset neural network. The preset neural network is used to generate a triangulation of the image to be processed. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional subnetwork and the second convolutional subnetwork respectively erode and dilate the image to be processed. The output layer is used to generate the triangulation based on the eroded image to be processed and the dilated image to be processed.
[0051] Fifthly, one embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for generating a tripartite graph as described in the first aspect or any implementation thereof, or implements a neural network construction method for generating a tripartite graph as described in the second aspect.
[0052] As one implementation of the fifth aspect, the processor includes a GPU or an NPU.
[0053] In a sixth aspect, one embodiment of this application provides a computer storage medium storing a computer program that, when executed by a processor, implements the method for generating a three-part graph as described in the first aspect or any implementation thereof.
[0054] As one implementation of the sixth aspect, the processor includes a GPU or an NPU.
[0055] In a seventh aspect, one embodiment of this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the method for generating a three-part graph as described in the first aspect or any implementation thereof, or to execute the neural network construction method for generating a three-part graph as described in the second aspect.
[0056] It should be understood that the beneficial effects of aspects three through seven can be found in the relevant descriptions of aspects one or two, and will not be repeated here. Attached Figure Description
[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0058] Figure 1 This is an example of a given image and its trisection in related technologies;
[0059] Figure 2 This is a schematic diagram illustrating the implementation process of a method for generating a three-part graph according to an embodiment of this application;
[0060] Figure 3 This is a schematic diagram of the structure of a neural model provided in an embodiment of this application;
[0061] Figure 4 This is a schematic diagram illustrating the implementation of step S120 of a method for generating a three-part image according to an embodiment of this application;
[0062] Figure 5 This is a schematic diagram of the structure of a first convolutional subnetwork provided in an embodiment of this application;
[0063] Figure 6 This is a schematic diagram of the structure of a second convolutional subnetwork provided in an embodiment of this application;
[0064] Figure 7 This is a schematic diagram of the structure of another neural model provided in one embodiment of this application;
[0065] Figure 8 This is a schematic diagram of an implementation process for normalizing an image to be processed, provided in an embodiment of this application;
[0066] Figure 9 This application provides a schematic diagram of the implementation process of a neural network construction method for generating triangulation maps according to an embodiment of the present application;
[0067] Figure 10 This is a schematic diagram of the structure of an apparatus for generating a three-part map according to an embodiment of this application;
[0068] Figure 11 This is a schematic diagram of the structure of a neural network construction device for generating a three-part graph according to an embodiment of this application;
[0069] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0070] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0071] The term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.
[0072] The terms "one embodiment" or "some embodiments" described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0073] Furthermore, in the description of this application, "a plurality of" means two or more. The terms "first" and "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0074] To illustrate the technical solution described in this application, specific embodiments are provided below.
[0075] Figure 2 This is a schematic diagram illustrating the implementation flow of a method for generating a three-part graph according to an embodiment of this application. The method for generating a three-part graph in this embodiment can be executed by an electronic device. Electronic devices include, but are not limited to, computers, tablets, servers, mobile phones, cameras, or wearable devices. Servers include, but are not limited to, dedicated servers or cloud servers. Figure 2 As shown, the method for generating a tripartite graph may include steps S110 to S120.
[0076] S110, Obtain the image to be processed.
[0077] The image to be processed can be an image or an image frame from a video sequence. The image to be processed can include a single-channel grayscale image or a binarized image. It should be noted that if the image to be processed is a color image, it needs to be segmented first to separate the image from the background, resulting in a binarized image.
[0078] In some embodiments, an image to be processed in a certain scene can be acquired by an acquisition device. The acquisition device acquires an image of a certain scene, the electronic device acquires the image, and then generates a tripartite image of the image. As one implementation, the acquisition device includes one or more combinations of cameras such as depth cameras, grayscale cameras, color cameras, or black and white cameras, etc., without limitation.
[0079] In some embodiments, the data acquisition device may be independent of the electronic device. In other embodiments, the data acquisition device may be integrated into the electronic device. This application does not limit this.
[0080] S120: Input the image to be processed into a preset neural network to generate a three-part image.
[0081] In some embodiments, a pre-defined neural network can be pre-deployed on the electronic device and stored in its memory for later use. In these embodiments, the process of building the pre-defined neural network can be performed by another device independent of the electronic device. After building the pre-defined neural network, the other device deploys it to the electronic device for use. It should be noted that the process of building the pre-defined neural network can be found later. Figure 9 The neural network construction method shown in the embodiment will not be described in detail here.
[0082] In some other embodiments, an initial neural network can be deployed in the electronic device beforehand. Before using the preset neural network, a preset neural network is constructed based on the initial neural network, and then used to generate a triangulation of the image to be processed. Specifically, in these embodiments, the process of constructing the preset neural network can be performed by the electronic device. Before the electronic device performs step S110 or S120, the step of constructing the preset neural network is also included. It should be noted that the construction process of the preset neural network can be found later. Figure 9 The neural network construction method shown in the embodiment will not be described in detail here.
[0083] In some embodiments, such as Figure 3 As shown, the preset neural network includes a backbone network 31 and an output layer 32 connected to the backbone network. The backbone network 31 includes a first convolutional sub-network 311 and a second convolutional sub-network 312 connected in parallel. The first convolutional network 311 and the second convolutional network 312 perform erosion and dilation on the image to be processed, respectively. The output layer 32 is used to generate a triangulation image based on the eroded and dilated images to be processed. Specifically, the output layer 32 includes a concatenated summation function and a preset region limitation function. The summation function adds the eroded and dilated images to obtain a summed image, and the preset region limitation function divides the summed image into three regions to generate the triangulation image.
[0084] As a non-restrictive example, such as Figure 4 As shown, step S120, which involves inputting the image to be processed into a preset neural network to generate a three-part image, may include steps S121 to S123.
[0085] S121, input the image to be processed into the first convolutional sub-network to obtain the erosion image; input the image to be processed into the second convolutional sub-network to obtain the dilation image.
[0086] S122, add the erosion image and the dilation image together to obtain the summed image.
[0087] S123, use a preset region limiting function to process the added image to obtain a trisection image, the region limiting function has three sub-regions.
[0088] As one implementation method, image erosion can be achieved through a simple network structure, such as... Figure 5 As shown, the first convolutional sub-network 311 includes a first convolutional layer 51, a second convolutional layer 52, and a first rounding layer 53. The first convolutional layer 51 uses a first convolutional kernel to filter the image to be processed; the second convolutional layer 52 adds a first bias to the output of the first convolutional layer 51, and the first bias is calculated based on the elements of the first convolutional kernel; the first rounding layer 53 rounds down the output of the second convolutional layer 52.
[0089] When the number of channels in the image to be processed is 1, the number of channels in the first convolution kernel can also be 1; when the number of channels in the image to be processed is 3, the number of channels in the first convolution kernel can also be 3. It should be understood that the exemplary description should not be construed as a limitation of this application.
[0090] Optionally, during the construction of the pre-defined neural network, the first convolutional kernel can be generated based on OpenCV's getStructuringElement function, with the specific calculation formula as follows:
[0091]
[0092] Where, k i It is the i-th element of the first convolution kernel. To generate the initial kernel corresponding to the first convolution kernel using the getStructuringElement function, i takes the value of an integer from 1 to N, where N is the total number of elements in the initial kernel.
[0093] It's important to understand that users or developers can specify the size and shape of the first convolutional kernel, and the `getStructuringElement` function can generate a first convolutional kernel of the specified shape and size. Similarly, electronic devices or development systems can specify the size and shape of the first convolutional kernel, and the `getStructuringElement` function can generate a first convolutional kernel of the specified shape and size. Furthermore, a first bias can be calculated based on the first convolutional kernel. The `getStructuringElement` function allows for the rapid generation of the first convolutional kernel, and the model can obtain optimal parameters without needing to learn them, improving model building efficiency and reducing costs. After generating the first convolutional kernel and calculating the first bias, both values can be assigned to the initial neural network; that is, the generated first convolutional kernel is assigned to the initial first convolutional kernel, and the calculated first bias is assigned to the initial first bias.
[0094] Optionally, the first bias_ero of the second convolutional layer 52 can be calculated according to the following formula:
[0095] bias_ero=-(A-thresh1*B)+A,
[0096] Where A and B are constants, thresh1 = max{k1,k2,…,k i}
[0097] Optionally, the first floor 53 may include a floor function.
[0098] As one implementation method, image dilation can be achieved through a simple network structure, such as... Figure 6As shown, the second convolutional sub-network 312 includes a third convolutional layer 61, a fourth convolutional layer 62, and a second rounding layer 63. The third convolutional layer 61 uses a second convolutional kernel to filter the image to be processed. The fourth convolutional layer 62 adds a second bias to the output of the third convolutional layer 61. The second bias is calculated based on the elements of the second convolutional kernel. The second rounding layer 63 rounds down the output of the fourth convolutional layer 62.
[0099] When the number of channels in the image to be processed is 1, the number of channels in the second convolution kernel can also be 1; when the number of channels in the image to be processed is 3, the number of channels in the second convolution kernel can also be 3. It should be understood that the exemplary description should not be construed as a limitation of this application.
[0100] Optionally, during the construction of the preset neural network, the second convolutional kernel is generated based on OpenCV's getStructuringElement function.
[0101] For example, a user or developer can specify the size and shape of the second convolutional kernel, and the `getStructuringElement` function can generate a second convolutional kernel with the specified shape and size. The specific calculation formula is as follows:
[0102]
[0103] Among them, p i The i-th element of the second convolution kernel. To generate the initial kernel corresponding to the second convolution kernel using the getStructuringElement function, the value of i is an integer from 1 to M, where M is the total number of elements in the initial kernel.
[0104] It should be noted that electronic devices or development systems can provide the size and shape of the second convolutional kernel. The `getStructuringElement` function can generate a second convolutional kernel of a specified shape and size. Furthermore, a second bias can be calculated based on the second convolutional kernel. The `getStructuringElement` function allows for rapid generation of the second convolutional kernel, and the model can obtain optimal parameters without needing to learn them, improving model building efficiency and reducing costs. After generating the second convolutional kernel and calculating the second bias, their values can be assigned to the initial neural network; that is, the generated second convolutional kernel is assigned to the initial second convolutional kernel, and the calculated second bias is assigned to the initial second bias.
[0105] It should be noted that in some embodiments, the size and shape of the first and second convolutional kernels can be the same. The `getStructuringElement` function can generate a first and second convolutional kernel of a specified shape and size simultaneously. In this case, the structure and parameters of the first convolutional layer 51 included in the first convolutional subnetwork 311 and the third convolutional layer 61 included in the second convolutional subnetwork 312 can be identical. Optionally, the first convolutional subnetwork 311 and the second convolutional subnetwork 312 can share the first convolutional layer, which can accelerate image processing, further improve processing efficiency, and save computing power.
[0106] Optionally, the second bias_dil of the fourth convolutional layer 62 is calculated according to the following formula:
[0107] bias_dil = -(thresh2*B) + A,
[0108] Where A and B are constants, thresh2 = max{p1, p2, ..., p i}
[0109] Optionally, the second floor 63 may include a floor function.
[0110] As one implementation, the output layer 32 includes a preset region limiting function called the `clamp` function. The region limiting function has three sub-regions. As a non-limiting example, the `clamp` function assigns a value of 2 to pixels in the summed image that are greater than 2, a value of 0 to pixels that are less than 0, and a value of 1 to pixels that are greater than 0 and less than 2, resulting in a trimap of 0, 1, or 2. Pixels in the summed image that are equal to 0 can be assigned a value of 0 or 1, and pixels in the summed image that are equal to 2 can be assigned a value of 1 or 2; this application does not impose any restrictions on this. It should be noted that the exemplary division of the three sub-regions here should not be construed as a specific limitation of this application.
[0111] In some other embodiments, in Figure 3 Based on the illustrated embodiment, the preset neural network further includes an input layer. For example... Figure 7 As shown, the preset neural network includes an input layer 30, a backbone network 31, and an output layer 32 connected to the backbone network. The input layer 30 preprocesses the image to be processed, for example, by normalizing the image to obtain a normalized image. The backbone network 31 performs erosion and dilation on the normalized image; specifically, the first convolutional subnetwork 311 erodes the normalized image, and the second convolutional subnetwork 312 dilates the normalized image. It should be understood that... Figure 7 and Figure 3 The similarities between the illustrated embodiments will not be repeated here; please refer to the foregoing.
[0112] It should be noted that normalizing the image to be processed not only makes the image conform to the input standard of the neural network, but also makes the image approximately uniformly distributed, thereby enhancing the contrast of the image and improving the accuracy of the triangulation.
[0113] As a non-restrictive example, such as Figure 8 As shown, the image to be processed is normalized to obtain a normalized image, including steps S810 and S820.
[0114] S810, determine the maximum and minimum values of each pixel in the image to be processed.
[0115] For the image to be processed, the maximum value of each pixel can be denoted as max_value, and the minimum value of each pixel can be denoted as min_value.
[0116] S820 normalizes each pixel in the image to be processed to the target value range by using the maximum and minimum values, thus obtaining a normalized image.
[0117] Specifically, each pixel in the image to be processed is normalized to the target numerical range [0,1]. For example, if the range of each pixel in the image to be processed is 0 to 255, the normalization operation will normalize these pixels to the range of 0 to 1.
[0118] For example, if the maximum value of each pixel in the image to be processed is max_value, and the minimum value of each pixel is min_value, then the pixel value of a certain pixel j in the image to be processed is Z. j Then the pixel Z of pixel j j Normalized to the target numerical range [0,1]:
[0119] Nor j =Z j / (max_value-min_value).
[0120] Among them, Nor j This represents the normalized value of pixel j.
[0121] It should be noted that the target value range can be set according to the actual situation, and this application does not impose specific restrictions.
[0122] In some other embodiments, in Figure 3 or Figure 7 Based on the illustrated embodiment, the preset neural network may include multiple cascaded first convolutional subnetworks 311 and / or multiple cascaded second convolutional subnetworks 312. That is, the backbone network 31 may contain multiple first convolutional subnetworks 311, and / or multiple second convolutional subnetworks 312. For example, the backbone network 31 may include three cascaded first convolutional subnetworks 311 and / or multiple second convolutional subnetworks 312. Figure 5 The first convolutional subnetwork 311 shown, the backbone network 31 also includes 3 cascaded subnetworks such as... Figure 6 The second convolutional subnetwork 312 shown adds the image after three dilations and the image after three erosions, and then divides it into three regions using a region constraint function to generate a tri-texture image. In these embodiments, multiple erosions and / or dilations are performed to increase the degree of erosion and / or dilation, thereby improving the stability of the tri-texture edges.
[0123] This application embodiment uses a neural network to erode and dilate the image to be processed, and then generates a triangulation based on the eroded and dilated image. The neural network can infer on a graphics processing unit (GPU) or a neural-network processing unit (NPU), which greatly improves the speed of generating triangulation.
[0124] Another embodiment of this application provides a neural network construction method for generating triangulation graphs. Where details are not described in detail in this embodiment, please refer to the foregoing embodiments; they will not be repeated here. Figure 9 As shown, the neural network construction method for generating a triangulation graph may include steps S910 to S940.
[0125] S910, construct the initial neural network for generating the triangulation.
[0126] In one embodiment, the initial neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional subnetwork and the second convolutional subnetwork erode and dilate the image to be processed, respectively. The output layer is used to generate a tripartite image based on the eroded image to be processed and the dilated image to be processed.
[0127] S920 determines the size and shape of the convolution kernel.
[0128] In some embodiments, the size and shape of the first convolutional kernel of the first convolutional subnetwork and the second convolutional kernel of the second convolutional subnetwork can be determined based on user or developer input data. In other embodiments, the size and shape of the first and second convolutional kernels can be determined based on system setting parameters.
[0129] The shapes of the first and second convolution kernels can include cross-shaped, elliptical, or rectangular shapes, and their sizes can be 3*3, 5*5, or 7*7, etc. This application does not impose specific restrictions on the shape and size.
[0130] S930 uses OpenCV's getStructuringElement function to generate elements of the convolution kernel and determines the bias based on the convolution kernel.
[0131] Specifically, the getStructuringElement function based on OpenCV generates elements of the first convolution kernel and the second convolution kernel respectively; the first bias is determined based on the first convolution kernel, and the second bias is determined based on the second convolution kernel.
[0132] S940 assigns the elements and biases of the convolution kernel to the initial neural network to obtain the preset neural network.
[0133] Specifically, the first convolution kernel and the first bias are assigned to the initial convolution kernel and the initial bias of the first convolutional subnetwork in the initial neural network, respectively, and the second convolution kernel and the second bias are assigned to the initial convolution kernel and the initial bias of the second convolutional subnetwork in the initial neural network, respectively, to obtain the preset neural network.
[0134] The preset neural network is used to generate a triangulation of the image to be processed. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional subnetwork and the second convolutional subnetwork erode and dilate the image to be processed, respectively. The output layer is used to generate a triangulation based on the eroded image to be processed and the dilated image to be processed.
[0135] It should be understood that the initial neural network and the preset neural network have the same structure, and the preset neural network is a network after some parameters of the initial neural network have been updated. Specifically, the preset neural network updates the four parameters of the initial neural network: the first convolution kernel, the second convolution kernel, the first bias, and the second bias.
[0136] The embodiments of this application generate a first convolutional kernel and a second convolutional kernel based on the getStructuringElement function, determine a first bias based on the first convolutional kernel, determine a second bias based on the second convolutional kernel, and then assign and update the four parameters of the first convolutional kernel, the second convolutional kernel, the first bias and the second bias in the initial neural network. This can greatly improve the construction efficiency of the neural network and greatly reduce the cost.
[0137] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0138] This application also provides an apparatus for generating a triangulation. For details not described in the apparatus, please refer to the relevant descriptions in the foregoing embodiments of the method for generating a triangulation.
[0139] See Figure 10 , Figure 10 This is a schematic block diagram of an apparatus for generating a three-part image according to an embodiment of this application. The apparatus for generating a three-part image may include an acquisition module 1010 and a generation module 1020.
[0140] The acquisition module 1010 is used to acquire the image to be processed;
[0141] The generation module 1020 is used to input the image to be processed into a preset neural network to generate a triangulation.
[0142] This application also provides a neural network construction apparatus for generating triangulations. For details not described in detail in this neural network construction apparatus for generating triangulations, please refer to the relevant descriptions in the foregoing embodiments of the neural network construction method for generating triangulations.
[0143] See Figure 11 , Figure 11 This is a schematic block diagram of a neural network construction apparatus for generating a triangulation graph according to an embodiment of this application. The neural network construction apparatus for generating a triangulation graph may include: a construction module 1100, a first determining module 1110, a second determining module 1120, and an assignment module 1130.
[0144] Module 1100 is used to build an initial neural network for generating the triangulation.
[0145] The first determining module 1110 is used to determine the size and shape of the convolution kernel of the initial neural network;
[0146] The second determining module 1120 is used to generate the elements of the convolution kernel based on the getStructuringElement function of OpenCV, and determine the bias according to the convolution kernel;
[0147] The assignment module 1130 is used to assign the elements and biases of the convolution kernel to the initial neural network to obtain the preset neural network.
[0148] The initial neural network has the same architecture as the preset neural network. The preset neural network is used to generate a triangulation of the image to be processed. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional network and the second convolutional subnetwork erode and dilate the image to be processed, respectively. The output layer is used to generate the triangulation based on the eroded image to be processed and the dilated image to be processed.
[0149] This application also provides an electronic device, such as... Figure 12As shown, the electronic device may include one or more processors 1200. Figure 12 (Only one is shown in the image), memory 1210, and computer program 1220 stored in memory 1210 and executable on one or more processors 1200, such as a program for generating a triangulation and / or constructing a neural network. When one or more processors 1200 execute computer program 1220, they can implement the various steps in the methods for generating a triangulation and / or constructing a neural network embodiments. Alternatively, when one or more processors 1200 execute computer program 1220, they can implement the functions of various modules / units in the apparatus for generating a triangulation and / or neural network construction apparatus embodiments, without limitation herein.
[0150] For example, computer program 1220 may be divided into one or more modules / units, one or more of which are stored in memory 1210 and executed by processor 1200 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 1220 in the processing unit.
[0151] For example, computer program 1220 can be divided into the following modules. The specific functions of each module are as follows:
[0152] The acquisition module is used to acquire the image to be processed;
[0153] The generation module is used to input the image to be processed into a preset neural network to generate a triangulation.
[0154] For example, computer program 1220 can be divided into the following modules. The specific functions of each module are as follows:
[0155] Build modules are used to construct the initial neural network for generating the triangulation.
[0156] The first determining module is used to determine the size and shape of the convolutional kernels of the initial neural network;
[0157] The second determining module is used to generate elements of the convolution kernel based on the getStructuringElement function of OpenCV, and to determine the bias according to the convolution kernel;
[0158] The assignment module is used to assign the elements and biases of the convolution kernel to the initial neural network to obtain the preset neural network.
[0159] Those skilled in the art will understand that Figure 12This is merely an example of an electronic device and does not constitute a limitation on electronic devices. Electronic devices may include more or fewer components than illustrated, or combinations of certain components, or different components. For example, electronic devices may also include input / output devices, network access devices, buses, etc.
[0160] In one embodiment, the processor 1200 may be a CPU, GPU, or NPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0161] In one embodiment, memory 1210 may be an internal storage unit of an electronic device, such as a hard disk or RAM. Memory 1210 may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 1210 may include both internal and external storage units. Memory 1210 is used to store computer programs and other programs and data required by the electronic device. Memory 1210 may also be used to temporarily store data that has been output or will be output.
[0162] This application also provides another preferred embodiment of an electronic device, in which the electronic device includes one or more processors. The one or more processors are used to execute the following program modules stored in memory:
[0163] The acquisition module is used to acquire the image to be processed;
[0164] The generation module is used to input the image to be processed into a preset neural network to generate a triangulation.
[0165] And / or,
[0166] Build modules are used to construct the initial neural network for generating the triangulation.
[0167] The first determining module is used to determine the size and shape of the convolutional kernels of the initial neural network;
[0168] The second determining module is used to generate elements of the convolution kernel based on the getStructuringElement function of OpenCV, and to determine the bias according to the convolution kernel;
[0169] The assignment module is used to assign the elements and biases of the convolution kernel to the initial neural network to obtain the preset neural network.
[0170] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0171] An embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the various steps in the embodiments of the method for generating a triangulation and / or the method for constructing a neural network.
[0172] An embodiment of this application also provides a computer program product that, when run on an electronic device, enables the electronic device to implement the various steps in the method and / or neural network construction method embodiment for generating a three-part graph.
[0173] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0174] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0175] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0178] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.
[0179] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for generating a tripartite graph, characterized in that, include: Obtain the image to be processed; The image to be processed is input into a preset neural network to generate a triangulation. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional subnetwork and the second convolutional subnetwork respectively erode and dilate the image to be processed. The output layer is used to generate the three-part image based on the eroded image to be processed and the dilated image to be processed, including: The image to be processed is input into the first convolutional subnetwork to obtain an erosion image; the image to be processed is input into the second convolutional subnetwork to obtain a dilation image; The erosion image and the dilation image are added together to obtain the summed image; The summed image is processed using a preset region limiting function to obtain the trisection image, wherein the region limiting function has three sub-regions; The first convolutional subnetwork and the second convolutional subnetwork each include a first convolutional layer, a second convolutional layer, and a rounding layer. The first convolutional layer uses a convolutional kernel to filter the image to be processed. The second convolutional layer adds a bias to the output of the first convolutional layer, and the bias is calculated based on the elements of the convolutional kernel. The rounding layer rounds down the output of the second convolutional layer.
2. The method for generating a tripartite graph as described in claim 1, characterized in that, The convolutional kernel is generated based on OpenCV's getStructuringElement function, and the calculation formula is as follows: in, For the i-th element of the convolution kernel, The i-th element of the initial generation kernel is generated using the getStructuringElement function, where i is an integer from 1 to j, and j is the total number of elements in the initial generation kernel.
3. The method for generating a tripartite graph as described in claim 2, characterized in that, For the first convolutional sub-network, the bias_ero is calculated according to the following formula: bias_ero = -(A - thresh * B) + A, Where A and B are constants, .
4. The method for generating a tripartite graph as described in claim 2, characterized in that, For the second convolutional sub-network, the bias_dil is calculated according to the following formula: bias_dil = -(thresh*B) + A Where A and B are constants, .
5. The method for generating a tripartite graph as described in claim 1, characterized in that, The preset neural network further includes an input layer, which normalizes the image to be processed to obtain a normalized image. The first convolutional subnetwork and the second convolutional subnetwork respectively erode and dilate the normalized image. The output layer is used to generate the tri-image based on the eroded normalized image and the dilated normalized image.
6. A method for constructing a neural network to generate triangulation maps, characterized in that, include: Construct an initial neural network for generating the triangulation; Determine the size and shape of the convolutional kernels of the initial neural network; The getStructuringElement function based on OpenCV generates the elements of the convolution kernel, and the bias is determined based on the convolution kernel; The elements and biases of the convolution kernel are assigned to the initial neural network to obtain the preset neural network; The initial neural network has the same architecture as the preset neural network. The preset neural network is used to generate a triangulation of the image to be processed. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The image to be processed is input into the first convolutional subnetwork to obtain an erosion image; the image to be processed is input into the second convolutional subnetwork to obtain a dilation image; The erosion image and the dilation image are added together to obtain the summed image; The summed image is processed using a preset region limiting function to obtain the trisection image, wherein the region limiting function has three sub-regions; The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional subnetwork and the second convolutional subnetwork respectively erode and dilate the image to be processed. The output layer is used to generate the three-part image based on the eroded image to be processed and the dilated image to be processed. The first convolutional subnetwork and the second convolutional subnetwork each include a first convolutional layer, a second convolutional layer, and a rounding layer. The first convolutional layer uses a convolutional kernel to filter the image to be processed. The second convolutional layer adds a bias to the output of the first convolutional layer, and the bias is calculated based on the elements of the convolutional kernel. The rounding layer rounds down the output of the second convolutional layer.
7. An apparatus for generating a tripartite graph, characterized in that, include: The acquisition module is used to acquire the image to be processed; A generation module is used to input the image to be processed into a preset neural network to generate a triangulation image. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional subnetwork and a second convolutional subnetwork connected in parallel. The first convolutional subnetwork and the second convolutional subnetwork respectively erode and dilate the image to be processed. The output layer is used to generate the triangulation image based on the eroded image to be processed and the dilated image to be processed. The module includes: The image to be processed is input into the first convolutional subnetwork to obtain an erosion image; the image to be processed is input into the second convolutional subnetwork to obtain a dilation image; The erosion image and the dilation image are added together to obtain the summed image; The summed image is processed using a preset region limiting function to obtain the trisection image, wherein the region limiting function has three sub-regions; The first convolutional subnetwork and the second convolutional subnetwork each include a first convolutional layer, a second convolutional layer, and a rounding layer. The first convolutional layer uses a convolutional kernel to filter the image to be processed. The second convolutional layer adds a bias to the output of the first convolutional layer, and the bias is calculated based on the elements of the convolutional kernel. The rounding layer rounds down the output of the second convolutional layer.
8. A neural network construction apparatus for generating triangulations, characterized in that, include: Build modules are used to construct the initial neural network for generating the triangulation. The first determining module is used to determine the size and shape of the convolution kernel of the initial neural network; The second determining module is used to generate the elements of the convolution kernel based on the getStructuringElement function of OpenCV, and determine the bias according to the convolution kernel; The assignment module is used to assign the elements and biases of the convolution kernel to the initial neural network to obtain the preset neural network; The initial neural network has the same architecture as the preset neural network. The preset neural network is used to generate a triangulation of the image to be processed. The preset neural network includes a backbone network and an output layer connected to the backbone network. The backbone network includes a first convolutional sub-network and a second convolutional sub-network connected in parallel. The first convolutional network and the second convolutional network respectively erode and dilate the image to be processed to obtain an eroded image and a dilated image. The eroded image and the dilated image are added together to obtain a summed image. The output layer is used to process the summed image based on the eroded image and the dilated image to be processed using a preset region limiting function to generate the triangulation. The region limiting function has three sub-regions. The first convolutional subnetwork and the second convolutional subnetwork each include a first convolutional layer, a second convolutional layer, and a rounding layer. The first convolutional layer uses a convolutional kernel to filter the image to be processed. The second convolutional layer adds a bias to the output of the first convolutional layer, and the bias is calculated based on the elements of the convolutional kernel. The rounding layer rounds down the output of the second convolutional layer.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating a triangulation as described in any one of claims 1 to 5, or implements the neural network construction method for generating a triangulation as described in claim 6.
10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for generating a triangulation as described in any one of claims 1 to 5, or implements the neural network construction method for generating a triangulation as described in claim 6.