Image data conversion method and device based on deep learning

Abstract

The present application provides a method, device, equipment and computer-readable storage medium for image data conversion based on deep learning. The method includes: obtaining knee joint mesh image data; inputting the knee joint mesh image data into a preset image data type conversion model, and outputting corresponding knee joint pixel image data; wherein the image data type conversion model is obtained through model training based on a fully convolutional neural network, and the fully convolutional neural network includes: 5 consecutive layers of convolutional layers, normalization layers and pooling layers, after cropping and fusion, the feature map is subjected to multi-level upsampling processing to obtain a feature matrix of the same size as the original image; Boolean calculation is performed on the knee joint pixel image data; and the knee joint pixel image data after the Boolean calculation is converted into the corresponding knee joint mesh image data. According to the embodiment of the present application, the accuracy of image data type conversion can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data conversion, and in particular, relates to a method, device, equipment and computer-readable storage medium for image data conversion based on deep learning. Background Art

[0002] After the knee joint mesh image data is cropped, it is easy for the operation of filling the missing area to fail to be completed successfully or the result is not as expected, that is, the probability of surface filling failure is relatively high. Figure 1 This is a comparison diagram of failed and successful surface patching provided by an embodiment of the present application. Because grid image data usually has complex geometric shapes, including various irregular boundaries and curved surfaces. During the clipping process, if the geometric shape is too complex, the surface patching algorithm may find it difficult to accurately infer the shape and attributes of the missing area, resulting in surface patching failure.

[0003] In order to reduce the probability of patching failure, the grid image data needs to be converted into pixel image data for processing and then converted back into grid image data.

[0004] Currently, the traditional method for image data type conversion is through a projection method, which projects a grid image onto a plane and then uses the projection result as a pixel image.

[0005] However, the image data method using the projection method has the following disadvantages:

[0006] Distortion: When projecting, the shape of the grid image may change, causing the transformed pixel image to be distorted from the original image. This may introduce additional errors and affect subsequent image processing and analysis tasks.

[0007] Information loss: Since projection methods usually need to map high-dimensional grid image data to a lower-dimensional pixel plane, some information may be lost. This may cause the converted pixel image to lack details or features in the original image, thus affecting the accuracy of subsequent analysis.

[0008] Therefore, how to improve the accuracy of image data type conversion is a technical problem that technical personnel in this field need to solve urgently. Summary of the invention

[0009] The embodiments of the present application provide a deep learning-based image data conversion method, apparatus, device, and computer-readable storage medium, which can improve the accuracy of image data type conversion.

[0010] In a first aspect, an embodiment of the present application provides an image data conversion method based on deep learning, comprising:

[0011] Acquire knee joint mesh image data;

[0012] Input the knee joint mesh image data into a preset image data type conversion model, and output the corresponding knee joint pixel image data; wherein the image data type conversion model is obtained through model training based on a fully convolutional neural network, and the fully convolutional neural network includes: 5 consecutive layers of convolutional layers, normalization layers, and pooling layers. After cropping and fusion, the feature map is subjected to multi-level upsampling processing to obtain a feature matrix of the same size as the original image;

[0013] Perform Boolean calculations on knee joint pixel image data;

[0014] The knee joint pixel image data after Boolean calculation is converted into corresponding knee joint mesh image data.

[0015] Optional, including:

[0016] Convolutional layer, used to extract image edge features and position features;

[0017] Normalization layer, used to ensure the stability of the weight distribution of the network after convolution;

[0018] The pooling layer is used to achieve feature dimensionality reduction and retain the main features extracted by the convolutional layer.

[0019] Optionally, perform Boolean calculations on the knee joint pixel image data, including:

[0020] Extract the features of joint contour and bone structure from the knee joint pixel image data;

[0021] Convert feature maps to binary images for Boolean computations.

[0022] Optionally, before extracting the features of the joint contour and the bone structure from the knee joint pixel image data, the method further includes:

[0023] Preprocess the pixel image data of the knee joint to extract clear feature information;

[0024] Among them, preprocessing includes denoising, smoothing, and enhancement operations.

[0025] Optionally, convert the feature map to a binary image for Boolean calculations, including:

[0026] According to the pixel distribution of the feature map, select an appropriate threshold to convert the pixel value into binary value;

[0027] Binarize the pixel values ​​in the feature map using the selected threshold;

[0028] According to the need, the binary image is filtered; wherein the filtering process includes: corrosion, expansion, opening operation, closing operation, so as to remove the noise or disconnected area;

[0029] According to the need, the converted binary image is subjected to Boolean calculation, wherein the Boolean calculation includes logical AND, logical OR, and logical NOT operations.

[0030] Optionally, binarize the pixel values ​​in the feature map using a selected threshold, including:

[0031] Each pixel is compared to the threshold, and pixels greater than the threshold are set to 1, and pixels less than or equal to the threshold are set to 0.

[0032] Optionally, converting the knee joint pixel image data after the Boolean calculation into corresponding knee joint mesh image data includes:

[0033] Determine the density and structure of the grid;

[0034] The knee joint pixel image data is converted into corresponding knee joint mesh image data through bilinear interpolation.

[0035] In a second aspect, an embodiment of the present application provides an image data conversion device based on deep learning, comprising:

[0036] A data acquisition module, used for acquiring mesh image data of the knee joint;

[0037] The first type conversion module is used to input the knee joint mesh image data into a preset image data type conversion model, and output the corresponding knee joint pixel image data; wherein the image data type conversion model is obtained through model training based on a fully convolutional neural network, and the fully convolutional neural network includes: a continuous 5-layer convolution layer, a normalization layer, and a pooling layer. After cropping and fusion, the feature map is subjected to multi-level upsampling processing to obtain a feature matrix of the same size as the original image;

[0038] A Boolean calculation module, used for performing Boolean calculation on the knee joint pixel image data;

[0039] The second type conversion module is used to convert the knee joint pixel image data after Boolean calculation into corresponding knee joint mesh image data.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0041] When the processor executes the computer program instructions, an image data conversion method based on deep learning is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, a deep learning-based image data conversion method is implemented.

[0043] The deep learning-based image data conversion method, device, equipment and computer-readable storage medium of the embodiments of the present application can improve the accuracy of image data type conversion.

[0044] The deep learning-based image data conversion method includes: acquiring knee joint grid image data; inputting the knee joint grid image data into a preset image data type conversion model, and outputting corresponding knee joint pixel image data; wherein the image data type conversion model is obtained through model training based on a fully convolutional neural network, and the fully convolutional neural network includes: 5 consecutive layers of convolutional layers, normalization layers, and pooling layers, and after cropping and fusion, multi-level upsampling processing is performed on the feature map to obtain a feature matrix with the same size as the original image; Boolean calculation is performed on the knee joint pixel image data; and the knee joint pixel image data after the Boolean calculation is converted into the corresponding knee joint grid image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 This is a schematic diagram comparing failed and successful surface repairs provided by an embodiment of the present application;

[0047] Figure 2 It is a flowchart of an image data conversion method based on deep learning provided by an embodiment of the present application;

[0048] Figure 3 is a structural diagram of an image data type conversion model provided by an embodiment of the present application;

[0049] Figure 4 is a structural schematic diagram of an image data conversion device based on deep learning provided by an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0052] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0053] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment and computer-readable storage medium for image data conversion based on deep learning. The following first introduces the method for image data conversion based on deep learning provided by the embodiments of the present application.

[0054] Figure 2 FIG. 1 is a flow chart of an image data conversion method based on deep learning provided by an embodiment of the present application. Figure 2 As shown, the image data conversion method based on deep learning includes:

[0055] S201, acquiring knee joint grid image data;

[0056] S202, inputting the knee joint mesh image data into a preset image data type conversion model, and outputting the corresponding knee joint pixel image data; wherein the image data type conversion model is obtained through model training based on a fully convolutional neural network, and the fully convolutional neural network includes: 5 consecutive layers of convolutional layers, normalization layers, and pooling layers, and after cropping and fusion, performing multi-level upsampling processing on the feature map to obtain a feature matrix of the same size as the original image;

[0057] S203, performing Boolean calculation on the knee joint pixel image data;

[0058] S204, converting the knee joint pixel image data after Boolean calculation into corresponding knee joint mesh image data.

[0059] Figure 3 : is a structural diagram of an image data type conversion model provided by an embodiment of the present application. In one embodiment, it includes:

[0060] Convolutional layer, used to extract image edge features and position features;

[0061] Normalization layer, used to ensure the stability of the weight distribution of the network after convolution;

[0062] The pooling layer is used to achieve feature dimensionality reduction and retain the main features extracted by the convolutional layer.

[0063] In one embodiment, a Boolean calculation is performed on the knee joint pixel image data, including:

[0064] Extract the features of joint contour and bone structure from the knee joint pixel image data;

[0065] Convert feature maps to binary images for Boolean computations.

[0066] In one embodiment, before extracting the features of the joint contour and the bone structure from the knee joint pixel image data, the method further includes:

[0067] Preprocess the pixel image data of the knee joint to extract clear feature information;

[0068] Among them, preprocessing includes denoising, smoothing, and enhancement operations.

[0069] In one embodiment, the feature map is converted into a binary image for Boolean calculation, including:

[0070] According to the pixel distribution of the feature map, select an appropriate threshold to convert the pixel value into binary value;

[0071] Binarize the pixel values ​​in the feature map using the selected threshold;

[0072] According to the need, the binary image is filtered; wherein the filtering process includes: corrosion, expansion, opening operation, closing operation, so as to remove the noise or disconnected area;

[0073] According to the need, the converted binary image is subjected to Boolean calculation, wherein the Boolean calculation includes logical AND, logical OR, and logical NOT operations.

[0074] In one embodiment, the pixel values ​​in the feature map are binarized using a selected threshold, including:

[0075] Each pixel is compared to the threshold, and pixels greater than the threshold are set to 1, and pixels less than or equal to the threshold are set to 0.

[0076] In one embodiment, converting the knee joint pixel image data after Boolean calculation into corresponding knee joint mesh image data includes:

[0077] Determine the density and structure of the grid;

[0078] The knee joint pixel image data is converted into corresponding knee joint mesh image data through bilinear interpolation.

[0079] Figure 4 : is a schematic diagram of the structure of an image data conversion device based on deep learning provided by an embodiment of the present application, the image data conversion device based on deep learning comprises:

[0080] A data acquisition module 401 is used to acquire knee joint mesh image data;

[0081] The first type conversion module 402 is used to input the knee joint mesh image data into a preset image data type conversion model, and output the corresponding knee joint pixel image data; wherein the image data type conversion model is obtained through model training based on a fully convolutional neural network, and the fully convolutional neural network includes: a continuous 5-layer convolution layer, a normalization layer, and a pooling layer, and after cropping and fusion, the feature map is subjected to multi-level upsampling processing to obtain a feature matrix with the same size as the original image;

[0082] Boolean calculation module 403, used for performing Boolean calculation on the knee joint pixel image data;

[0083] The second type conversion module 404 is used to convert the knee joint pixel image data after Boolean calculation into corresponding knee joint mesh image data.

[0084] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.

[0085] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0086] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0087] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 502 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 502 may be inside or outside the electronic device. In a particular embodiment, the memory 502 may be a non-volatile solid-state memory.

[0088] In one embodiment, the memory 502 may be a read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0089] The processor 501 implements any one of the deep learning-based image data conversion methods in the above embodiments by reading and executing computer program instructions stored in the memory 502 .

[0090] In one example, the electronic device may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0091] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0092] Bus 510 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0093] In addition, in combination with the image data conversion method based on deep learning in the above embodiments, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the image data conversion methods based on deep learning in the above embodiments is implemented.

[0094] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0095] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0096] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0097] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0098] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for image data conversion based on deep learning, characterized in that: include: Acquire knee joint mesh image data; Input the knee joint mesh image data into a preset image data type conversion model, and output the corresponding knee joint pixel image data; wherein the image data type conversion model is obtained through model training based on a fully convolutional neural network, and the fully convolutional neural network includes: 5 consecutive layers of convolutional layers, normalization layers, and pooling layers. After cropping and fusion, the feature map is subjected to multi-level upsampling processing to obtain a feature matrix of the same size as the original image; Perform Boolean calculations on knee joint pixel image data; Convert the knee joint pixel image data after Boolean calculation into corresponding knee joint mesh image data; Perform Boolean calculations on knee joint pixel image data, including: Extract the features of joint contour and bone structure from the knee joint pixel image data; Convert feature maps into binary images for Boolean calculations; Before extracting the features of the joint contour and bone structure from the knee joint pixel image data, it also includes: Preprocess the pixel image data of the knee joint to extract clear feature information; Among them, preprocessing includes denoising, smoothing, and enhancement operations; Convert feature maps to binary images for Boolean computations, including: According to the pixel distribution of the feature map, select an appropriate threshold to convert the pixel value into binary value; Binarize the pixel values ​​in the feature map using the selected threshold; According to the need, the binary image is filtered; wherein the filtering process includes: corrosion, expansion, opening operation, closing operation, so as to remove the noise or disconnected area; According to the need, the converted binary image is subjected to Boolean calculation, wherein the Boolean calculation includes logical AND, logical OR, and logical NOT operations; Binarize the pixel values ​​in the feature map using the selected threshold, including: Each pixel is compared to the threshold, and pixels greater than the threshold are set to 1, and pixels less than or equal to the threshold are set to 0.

2. The image data conversion method based on deep learning according to claim 1, characterized in that: include: Convolutional layer, used to extract image edge features and position features; Normalization layer, used to ensure the stability of the weight distribution of the network after convolution; The pooling layer is used to achieve feature dimensionality reduction and retain the main features extracted by the convolutional layer.

3. The image data conversion method based on deep learning according to claim 1, characterized in that: Convert the knee joint pixel image data after Boolean calculation into the corresponding knee joint mesh image data, including: Determine the density and structure of the grid; The knee joint pixel image data is converted into corresponding knee joint mesh image data through bilinear interpolation.

4. An image data conversion device based on deep learning, characterized in that: The device comprises: A data acquisition module, used for acquiring mesh image data of the knee joint; The first type conversion module is used to input the knee joint mesh image data into a preset image data type conversion model, and output the corresponding knee joint pixel image data; wherein the image data type conversion model is obtained through model training based on a fully convolutional neural network, and the fully convolutional neural network includes: a continuous 5-layer convolution layer, a normalization layer, and a pooling layer. After cropping and fusion, the feature map is subjected to multi-level upsampling processing to obtain a feature matrix of the same size as the original image; A Boolean calculation module, used for performing Boolean calculation on the knee joint pixel image data; The second type conversion module is used to convert the knee joint pixel image data after Boolean calculation into the corresponding knee joint mesh image data; Perform Boolean calculations on knee joint pixel image data, including: Extract the features of joint contour and bone structure from the knee joint pixel image data; Convert feature maps into binary images for Boolean calculations; Before extracting the features of the joint contour and bone structure from the knee joint pixel image data, it also includes: Preprocess the pixel image data of the knee joint to extract clear feature information; Among them, preprocessing includes denoising, smoothing, and enhancement operations; Convert feature maps to binary images for Boolean computations, including: According to the pixel distribution of the feature map, select an appropriate threshold to convert the pixel value into binary value; Binarize the pixel values ​​in the feature map using the selected threshold; According to the need, the binary image is filtered; wherein the filtering process includes: corrosion, expansion, opening operation, closing operation, so as to remove the noise or disconnected area; According to the need, the converted binary image is subjected to Boolean calculation, wherein the Boolean calculation includes logical AND, logical OR, and logical NOT operations; Binarize the pixel values ​​in the feature map using the selected threshold, including: Each pixel is compared to the threshold, and pixels greater than the threshold are set to 1, and pixels less than or equal to the threshold are set to 0.

5. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the deep learning-based image data conversion method as described in any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the deep learning-based image data conversion method as described in any one of claims 1 to 3.

Images