Image data processing method and device

Abstract

The invention relates to an image data processing method and device. The method comprises the steps of collecting first image data meeting a first preset condition and second image data meeting a second preset condition; the first image data corresponds to a part of pixel points in the first image, and the second image data corresponds to a part of pixel points in the second image; caching the first image data and the second image data to a first storage unit and a second storage unit respectively; under the condition that the data volume of the first target image data meets the data volume condition, pixel point interpolation is carried out on the first target image data to obtain third image data; and encoding the third image data and the second target image data to obtain encoded image data. By adopting the method, the image data processing efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image data processing method and apparatus. Background Technology

[0002] With the continuous development of electronic technology, endoscopic imaging systems have emerged, which are widely used for the observation of internal organs and the diagnosis of diseases. Endoscopic imaging systems utilize CMOS sensors in conjunction with a cold light source to acquire image data of internal organs, and then cache, upload, and display the received image data from the CMOS sensor.

[0003] With the continuous improvement of medical technology, the field of view of a single-lens endoscope has become relatively limited. As a result, dual-lens endoscope imaging systems have emerged. However, current dual-lens endoscope imaging systems have low real-time image display due to the large amount of image data they need to process. Obviously, endoscopic images with low real-time performance are not conducive to improving diagnostic accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide an image data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of image data processing in order to address the above-mentioned technical problems.

[0005] In a first aspect, this application provides an image data processing method, including:

[0006] Collect first image data that meets a first preset condition and second image data that meets a second preset condition; the first image data corresponds to a portion of the pixels in the first image, and the second image data corresponds to a portion of the pixels in the second image;

[0007] The first image data and the second image data are cached in the first storage unit and the second storage unit, respectively;

[0008] If the amount of data in the first target image data meets the data amount condition, pixel interpolation is performed on the first target image data to obtain the third image data. The first target image data is the image data with the smaller corresponding image size between the first image data and the second image data.

[0009] The third image data and the second target image data are encoded to obtain encoded image data. The second target image data is the image data other than the first target image data in the first image data and the second image data.

[0010] Secondly, this application also provides an image data processing apparatus, comprising:

[0011] The acquisition module is used to acquire first image data that meets a first preset condition and second image data that meets a second preset condition; the first image data corresponds to a portion of pixels in the first image, and the second image data corresponds to a portion of pixels in the second image;

[0012] A caching module is used to cache the first image data and the second image data into the first storage unit and the second storage unit, respectively;

[0013] The interpolation module is used to perform pixel interpolation on the first target image data to obtain the third image data when the data volume of the first target image data meets the data volume condition. The first target image data is the image data with the smaller corresponding image size between the first image data and the second image data.

[0014] The encoding module is used to encode the third image data and the second target image data to obtain encoded image data. The second target image data is the image data other than the first target image data in the first image data and the second image data.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement some or all of the steps described in any method of the first aspect of the embodiments of this application.

[0016] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements some or all of the steps described in any method of the first aspect of the embodiments of this application.

[0017] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements some or all of the steps described in any method of the first aspect of the embodiments of this application.

[0018] The aforementioned image data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire first image data that meets a first preset condition and second image data that meets a second preset condition; the first image data corresponds to a portion of pixels in the first image, and the second image data corresponds to a portion of pixels in the second image; the first image data and the second image data are respectively cached in a first storage unit and a second storage unit; when the data volume of the first target image data meets the data volume condition, pixel interpolation is performed on the first target image data to obtain third image data, where the first target image data is the image data with the smaller corresponding image size between the first image data and the second image data; the third image data and the second target image data are encoded to obtain encoded image data, where the second target image data is the image data between the first image data and the second image data excluding the first target image data. The image data processing method provided in this embodiment can encode the third image data and the second target image data obtained by pixel interpolation of the first target image data to obtain encoded image data without waiting for all pixels of the first image and the second image to be acquired. Based on this, there is no need to cache the complete first image and the second image, but only cache a portion of the pixels in the first image and a portion of the pixels in the second image, which significantly improves the efficiency of image data processing. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is an application environment diagram of an image data processing method in one embodiment;

[0021] Figure 2 This is a flowchart illustrating an image data processing method in one embodiment;

[0022] Figure 3 This is a schematic diagram of the endoscope structure in one embodiment;

[0023] Figure 4 This is a structural block diagram of an image data processing device in one embodiment;

[0024] Figure 5 This is an internal structural diagram of a computer device in one embodiment;

[0025] Figure 6This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] The image data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the host computer 102 communicates with the medical device 104 via a network. A data storage system can store the data that the medical device 104 needs to process. The data storage system can be integrated into the medical device 104 or placed in the cloud or on another network server. The medical device 104 is a device with two camera devices; for example, it can be a dual-lens endoscope with two camera devices, or other devices with two camera devices, such as an ultrasonic endoscope with two camera devices. There are no limitations on this.

[0028] The host computer 102 can be a smartphone, tablet computer, laptop computer, desktop computer, etc.

[0029] The host computer 102 and the medical device 104 can be connected via Bluetooth, USB (Universal Serial Bus) or network communication, and this application does not impose any restrictions on this.

[0030] In one exemplary embodiment, such as Figure 2 As shown, an image data processing method is provided, which is applied to... Figure 1 The following steps are used as an example of medical equipment, including steps 202 to 208.

[0031] in:

[0032] Step 202: Collect first image data that meets the first preset condition and second image data that meets the second preset condition; the first image data corresponds to a portion of the pixels in the first image, and the second image data corresponds to a portion of the pixels in the second image.

[0033] The first preset condition defines the image data corresponding to a specific region in the first image that needs to be collected. In simple terms, the image data corresponding to the specific region in the first image includes the first image data. Correspondingly, the second preset condition defines the image data corresponding to a specific region in the second image that needs to be collected. In simple terms, the image data corresponding to the specific region in the second image includes the second image data.

[0034] Since the first image and the second image were acquired by medical equipment, the first image and the second image are medical images, and correspondingly, the first image data and the second image data are medical image data.

[0035] The size of the first image is different from that of the second image, and correspondingly, the amount of data in the first image is different from that in the second image.

[0036] The first image data corresponds to a portion of the pixels in the first image, and the second image data corresponds to a portion of the pixels in the second image. That is to say, it is not necessary to wait until all the pixels in the first image and all the pixels in the second image are collected before step 202 can be completed and step 204 can be executed.

[0037] Optionally, the first image and the second image can be rectangular images or square images; if the first image and the second image are rectangular images, the aspect ratio of the first image and the aspect ratio of the second image are the same.

[0038] Alternatively, the medical device may be an endoscope, and more specifically, the endoscope may be a dual-lens endoscope with two different camera devices.

[0039] In an exemplary embodiment, the acquisition of first image data satisfying a first preset condition and second image data satisfying a second preset condition includes:

[0040] Collect first initial image data that meets the first preset condition and second initial image data that meets the second preset condition; perform image preprocessing on the first initial image data and the second initial image data respectively to obtain the first image data and the second image data respectively.

[0041] Image preprocessing includes black level compensation, depigmentation, white balance adjustment, and image denoising. It is easy to understand that the first and second image data obtained after image preprocessing have high image quality, ensuring that the subsequently obtained third and second target image data also have high image quality.

[0042] Step 204: Cache the first image data and the second image data into the first storage unit and the second storage unit, respectively.

[0043] The first storage unit is used to cache the first image data, and the second storage unit is used to cache the second image data. That is to say, the first storage unit corresponds to the first image data, and the second storage unit corresponds to the second image data.

[0044] The steps of caching the first image data into the first storage unit and caching the second image data into the second storage unit are performed simultaneously. That is to say, it is not necessary to wait until the first image data is cached into the first storage unit before caching the second image data into the second storage unit, nor is it necessary to wait until the second image data is cached into the second storage unit before caching the first image data into the first storage unit.

[0045] Since the first image data corresponds to a portion of the pixels in the first image and the second image data corresponds to a portion of the pixels in the second image, the first storage unit does not need to cache all the pixels in the first image, and the second storage unit does not need to cache all the pixels in the second image.

[0046] Step 206: If the amount of data in the first target image data meets the data amount condition, perform pixel interpolation on the first target image data to obtain the third image data. The first target image data is the image data with the smaller corresponding image size between the first image data and the second image data.

[0047] The data volume of the first target image data refers to the number of pixels in the first target image data.

[0048] Optionally, the first target image data can be interpolated pixel by pixel using a bilinear interpolation algorithm to obtain the third image data.

[0049] The first target image data is the image data with the smaller corresponding image size between the first image data and the second image data. That is to say, if the size of the first image corresponding to the first image data is smaller than the size of the second image corresponding to the second image data, then the first target image data is the first image data. Conversely, if the size of the second image corresponding to the second image data is smaller than the size of the first image corresponding to the first image data, then the first target image data is the second image data.

[0050] Optionally, the image data with the larger image size in the first image data and the second image data may not undergo pixel interpolation processing, or it may undergo pixel interpolation to obtain image data with a larger number of pixels.

[0051] Step 208: Encode the third image data and the second target image data to obtain encoded image data. The second target image data is the image data other than the first target image data in the first image data and the second image data.

[0052] Encoding the third image data and the second target image data to obtain encoded image data is intended to enable the medical device to send the encoded image data to the host computer via only one data interface in subsequent steps, instead of sending the third image data and the second target image data to the host computer separately through multiple data interfaces. This significantly reduces the power consumption of the medical device and further improves the efficiency of image data processing.

[0053] Optionally, the third image data and the second target image data are encoded using a joint encoding method to obtain encoded image data including a header, a trailer, and a data body, with the data body located between the header and the trailer. The header in the encoded image data identifies the start of the encoded image data and typically contains key information such as data type, data length, and checksum. The trailer identifies the end of the encoded image data to ensure its integrity and correctness. The data body stores the third image data and the second target image data. Therefore, if the medical device sends the encoded image data to a host computer, the host computer, upon receiving the encoded image data, can accurately decode the encoded image data based on the header and trailer to obtain the third image data and the second target image data.

[0054] The second target image data is the image data with the larger corresponding image size between the first image data and the second image data.

[0055] Optionally, the second target image data can be image data other than the first target image data in the first image data and the second image data, or it can be image data after pixel interpolation of the image data other than the first target image data in the first image data and the second image data.

[0056] Optionally, the medical device may include a Field-Programmable Gate Array (FPGA) as the core processor of the medical device, a first camera device, a second camera device, and a serial expansion bus interface. The FPGA includes a first memory unit and a second memory unit. The FPGA can be used to determine whether the amount of image data in different memory units meets the data volume requirements. The FPGA can also perform pixel interpolation, image encoding processing, and other tasks.

[0057] It is easy to understand that the third image data corresponds to the third image, the second target image data corresponds to the fourth image, and the third and fourth images correspond to either the first or the second image, respectively. If the third image corresponds to the first image, then the fourth image corresponds to the second image, and vice versa. It is easy to understand that the third image data and the second target image data are medical image data, and the third and fourth images are medical images.

[0058] In an exemplary embodiment, a first camera device and a second camera device respectively acquire first image data that meets a first preset condition and second image data that meets a second preset condition; the FPGA caches the first image data and the second image data into a first storage unit and a second storage unit respectively; when the data volume of the first target image data meets the data volume condition, the FPGA performs pixel interpolation on the first target image data to obtain third image data, wherein the first target image data is the image data with the smaller corresponding image size between the first image data and the second image data; the FPGA encodes the third image data and the second target image data to obtain encoded image data, wherein the second target image data is the image data between the first image data and the second image data excluding the first target image data.

[0059] In the above image data processing method, first image data satisfying a first preset condition and second image data satisfying a second preset condition are acquired; the first image data corresponds to a portion of pixels in the first image, and the second image data corresponds to a portion of pixels in the second image; the first image data and the second image data are cached in the first storage unit and the second storage unit, respectively; when the data volume of the first target image data meets the data volume condition, pixel interpolation is performed on the first target image data to obtain third image data, where the first target image data is the image data with the smaller corresponding image size between the first image data and the second image data; the third image data and the second target image data are encoded to obtain encoded image data, where the second target image data is the image data in the first image data and the second image data excluding the first target image data. Using the image data processing method provided in this embodiment, it is not necessary to wait until all pixels of the first image and all pixels of the second image are acquired before encoding the third image data and the second target image data obtained by pixel interpolation of the first target image data to obtain encoded image data. Therefore, it is not necessary to cache the complete first and second images, but only a portion of pixels in the first image and a portion of pixels in the second image are cached, significantly improving the image data processing efficiency.

[0060] In one exemplary embodiment, the data volume condition is two rows of pixels.

[0061] In this embodiment, the data volume condition is set to two rows of pixels. This allows the medical device to perform pixel interpolation on the first target image data to obtain the third image data when the data volume of the first target image data reaches two rows of pixels. On the one hand, this avoids the defect that the image data after pixel interpolation is less accurate due to the lack of sufficient image context information in a single row of pixels. On the other hand, it allows the medical device to perform pixel interpolation based on only two rows of pixels, without having to wait for more or even all pixels to be collected before performing pixel interpolation. This significantly improves the efficiency of image data processing.

[0062] In an exemplary embodiment, the above-described pixel interpolation of the first target image data to obtain the third image data includes:

[0063] Determine the 2×2 pixel matrix in the first target image data.

[0064] Pixel interpolation is performed on each pixel matrix to obtain the third image data.

[0065] Specifically, when the data volume condition is two rows of pixels, i.e., the first target image data consists of two rows of pixels, the first target image data includes multiple 2×2 pixel matrices.

[0066] In the process of interpolating each pixel matrix in the first target image data to obtain the third image data, the number of pixels that each pixel matrix needs to be interpolated to is determined by a first preset scaling factor, which is a multiple between the number of pixels in the third image data and the number of pixels in the first target image data.

[0067] Alternatively, pixel-by-pixel interpolation can be performed on each pixel matrix based on a bilinear interpolation algorithm to obtain the third image data.

[0068] For example, when the first target image data includes two rows and one hundred columns of pixels, the first target image data includes a pixel matrix composed of the first column of pixels and the second column of pixels, a pixel matrix composed of the second column of pixels and the third column of pixels, a pixel matrix composed of the third column of pixels and the fourth column of pixels, ... a pixel matrix composed of the ninety-ninth column of pixels and the one hundredth column of pixels.

[0069] For example, assuming the size of the third image data is 6×6 pixels, the specific process of interpolating each 2×2 pixel matrix using the bilinear interpolation algorithm to obtain the third image data is as follows: The coordinates of the four pixels in the pixel matrix are determined as (0, 0), (0, 1), (1, 0), and (1, 1), respectively. The pixels in the third image data are represented as (x, y). The two known pixels on the line passing through pixel (x, y) are represented as (x0, y0) and (x1, y1). When the 2×2 pixel matrix is ​​enlarged to a 6×6 pixel matrix, the number of pixels to be interpolated is 6×6 - 2×2 = 32. The ordinates of the 32 pixels (x, y) to be interpolated can be represented by the following formula:

[0070]

[0071] It should be noted that FPGAs in medical devices typically do not support direct floating-point arithmetic or have low efficiency in floating-point operations. Therefore, if floating-point calculations are involved in pixel interpolation, the floating-point data needs to be converted to fixed-point data for processing; that is, fixed-point conversion is required. Fixed-point conversion is a method of mapping floating-point numbers to integers by multiplying by a fixed quantization factor.

[0072] In an exemplary embodiment, the above-mentioned pixel interpolation of each pixel matrix to obtain third image data includes: performing pixel interpolation on each pixel matrix, and performing fixed-point processing on the pixel-interpolated image data to obtain third image data; the quantization coefficient in the fixed-point processing corresponds to the first preset scaling factor.

[0073] Similarly, the process of obtaining the second target image data also requires quantization. The quantization coefficients involved in obtaining the second target image data correspond to the second preset scaling factor, which will not be elaborated on in the following text.

[0074] In this embodiment, a 2×2 pixel matrix is ​​determined in the first target image data, and then pixel interpolation is performed on each pixel matrix to obtain the third image data. Thus, by generating more pixels in the first target image data, the number of pixels in the first target image data is increased, so that the third image data after pixel interpolation of the first target image data can have a clearer visual effect, and the image details of the third image data are richer to more accurately reflect the features and structure of the observed object. Based on this, medical staff can make more accurate diagnostic results based on the third image corresponding to the third image data.

[0075] In one exemplary embodiment, the method further includes:

[0076] The encoded image data is sent to the host computer so that the host computer can decode the encoded image data to obtain the third image data and the second target image data.

[0077] The method also includes:

[0078] Receive the first and second preset conditions from the host computer.

[0079] In this process, after the host computer decodes the encoded image data to obtain the third image data and the second target image data, the host computer generates a partial third image based on the third image data and pushes it to the display interface. Simultaneously, the host computer generates a partial fourth image based on the second target image data and pushes it to the display interface. The partial third image corresponds to the third image data, and the partial fourth image corresponds to the second target image data.

[0080] The medical device receives first and second preset conditions from the host computer, thereby enabling the medical device to perform image data acquisition based on the first and second preset conditions, that is, to acquire first image data that meets the first preset condition and second image data that meets the second preset condition.

[0081] Optionally, the first preset condition and the second preset condition can be set separately by medical staff on the host computer. That is to say, the first preset condition and the second preset condition can be flexibly set according to the visual needs of medical staff for medical images.

[0082] Since the first image data and the second image data correspond to a portion of the pixels in the first image and the second image, respectively, the third image data and the second target image data also correspond to a portion of the pixels in the third image and the fourth image, respectively.

[0083] In a straightforward manner, since the medical device continuously sends the third image data and the second target image data based on the image row data acquisition and pixel interpolation processing to the host computer, the host computer can also continuously display the third and fourth images of the image row data. Based on this, the host computer does not need to wait until a complete frame of image can be displayed before displaying the complete image frame. Thus, the host computer can achieve real-time image display.

[0084] In this embodiment, the medical device receives first and second preset conditions from the host computer. Based on these conditions, the medical device can acquire image data. Furthermore, the medical device sends the encoded image data, obtained after pixel interpolation and image encoding, to the host computer. This allows the host computer to decode the encoded image data and display the corresponding portions of the third and fourth images. Therefore, this embodiment can achieve real-time display of the scaled portions of the third and fourth images without waiting for the medical device to complete image data processing for all third and fourth images. This real-time image data processing enables real-time image display, allowing medical personnel to make more accurate diagnostic conclusions based on the displayed images with guaranteed real-time performance.

[0085] In an exemplary embodiment, the above-described pixel interpolation of the first target image data to obtain the third image data includes:

[0086] The first target image data is interpolated pixel by pixel based on the first preset scaling factor to obtain the third image data.

[0087] The first preset scaling factor refers to the area scaling factor between the third image data and the first target image data.

[0088] It is easy to understand that when the first preset scaling factor is greater than 0, the size of the image corresponding to the third image data is greater than the size of the image corresponding to the first target image data, that is, the image data is enlarged. The second preset scaling factor is similar and will not be described again below.

[0089] For example, if the size of the first target image data is 2×2 and the size of the third image data is 6×6, then the first preset scaling factor = (6 / 2)×(6 / 2) = 9.

[0090] In an exemplary embodiment, when the image size corresponding to the third image data is larger than the image size corresponding to the second target image data, the above method further includes:

[0091] If the amount of data in the larger image data in the first image data and the second image data meets the data amount condition, pixel interpolation is performed on the larger image data in the first image data and the second image data based on the second preset scaling factor to obtain the second target image data.

[0092] The data volume of the second image data refers to the number of pixels in the second image data.

[0093] Optionally, pixel-level interpolation can be performed on the second image data based on a bilinear interpolation algorithm to obtain the second target image data.

[0094] The first preset scaling factor corresponds to the image data with the smaller image size in the first image data and the second image data, and the second preset scaling factor corresponds to the image data with the larger image size in the first image data and the second image data. For example, when the image size corresponding to the first image data is smaller than the image size corresponding to the second image data, the first preset scaling factor corresponds to the first image data, and the second preset scaling factor corresponds to the second image data.

[0095] In this embodiment, pixel interpolation is performed on the first target image data and the second target image data based on the first preset scaling factor and the second preset scaling factor. This enables accurate scaling of the first image data and the second image data, thereby improving the accuracy of image data processing and ensuring that the host computer can obtain high-quality third image data and second target image data after decoding the encoded image data.

[0096] It is easy to understand that when both the first preset scaling factor and the second preset scaling factor are greater than 0, that is, when both the first image data and the second image data are processed by pixel interpolation to obtain image size enlargement, assuming that the third image data corresponds to the first image data and the second target image data corresponds to the second image data, then the number of pixels in the third image data is greater than the number of pixels in the first image data, and the number of pixels in the second target image data is greater than the number of data points in the second image data. That is, the number of pixels in the third image data and the number of pixels in the second target image data are both more than two rows of pixels.

[0097] In an exemplary embodiment, the method further includes receiving a first preset scaling factor and a second preset scaling factor from a host computer.

[0098] In one exemplary embodiment, the first preset condition includes a first target area.

[0099] The second preset condition includes the second target area.

[0100] The first target region can be the image data in the first image that needs to be scaled, specifically the start row, start column, end row, and end column of the first image that need to be scaled. Similarly, the second target region can be the image data in the second image that needs to be scaled, specifically the start row, start column, end row, and end column of the second image that need to be scaled.

[0101] In this embodiment, the first preset condition and the second preset condition respectively include the first target area and the second target area. Thus, the medical device can accurately collect the image data that needs to be scaled in the first image, i.e., the first image data, and the image data that needs to be scaled in the second image, i.e., the second image data. The image data that does not need to be scaled is not cached. In this way, the accuracy of image data processing is improved by improving the targeting of image data acquisition.

[0102] In one exemplary embodiment, the first image and the second image are images captured by the endoscope through dual cameras, respectively.

[0103] The endoscope is a dual-lens endoscope comprising a first camera device and a second camera device, with the first and second camera devices located at different positions within the endoscope. First image data is acquired through the first camera device, and second image data is acquired through the second camera device. The first camera device corresponds to a first field of view, and the second camera device corresponds to a second field of view.

[0104] Optionally, there may be an overlapping field of view between the first field of view of the first camera device and the second field of view of the second camera device. That is, there may be an intersection of the fields of view between the first camera device and the second camera device, so that there is an overlapping field of view between the first field of view corresponding to the first image data and the second field of view corresponding to the second image data.

[0105] Since the first camera device and the second camera device are located at different positions on the endoscope, there is a certain distance between the optical centers of the first camera device and the second camera device, and there is a certain angle between the optical axes of the first camera device and the second camera device.

[0106] The distance between the optical center of the first camera device and the optical center of the second camera device is greater than 0.

[0107] The angle between the optical axis of the first camera device and the optical axis of the second camera device is greater than 0° and less than or equal to 90°, that is, the angle range is 0 to 90°.

[0108] It should be noted that by performing a Fourier transform on an image, it can be converted from the spatial domain to the frequency domain. Frequency domain to spatial domain conversion: Through the Fourier transform, an image can be converted from the spatial domain to the frequency domain. Thus, the low-frequency components of an image correspond to slowly changing regions (e.g., the background area, large uniform areas), while the high-frequency components correspond to rapidly changing regions (e.g., edges, textures).

[0109] Optionally, slowly changing and rapidly changing image regions can be distinguished by the rate of change of grayscale values. Thus, slowly changing image regions have smoother grayscale transitions, while rapidly changing image regions experience more drastic fluctuations in grayscale values. In other words, whether an image region is a high-frequency or low-frequency component is determined by the image's spatial frequency, which is represented by the rate of change of grayscale values ​​in space. Based on this, optionally, high-frequency components can refer to image regions where the grayscale change rate is greater than or equal to a preset change rate, and low-frequency components can refer to image regions where the grayscale change rate is less than a preset change rate.

[0110] Optionally, the first image is dominated by low-frequency components, and the second image is dominated by high-frequency components. That is, in the first image and the second image, the first image may be dominated by slowly changing image regions, and the second image may be dominated by rapidly changing image regions.

[0111] When the first image is dominated by low-frequency components, the first image has richer global information. That is to say, the first image can better present the overall structure of the target object and the large range of grayscale changes. For example, based on the first image dominated by low-frequency components, the overall outline of the stomach, the approximate location and size of each anatomical region, and other global information can be clearly seen.

[0112] When the second image is dominated by high-frequency components, it possesses richer local information. That is, the second image exhibits higher contrast in biological tissues such as blood vessels and mucous membranes, making it easier to capture the details of these tissues. For example, the details of blood vessels may include texture information such as the direction of the blood vessels, their branches, and the texture of the vessel walls; the details of mucous membranes may include texture information such as the glandular structures of the mucous membrane.

[0113] Optionally, such as Figure 3 As shown, the endoscope 30 is an ultrasonic endoscope. The endoscope 30 includes a first camera device 302 for acquiring a first image, a second camera device 304 for acquiring a second image, and an ultrasonic probe 306. The first camera device 302 is located in the forward viewing direction of the endoscope 30, which is also the visual reference direction used to guide the user's operation of the endoscope 30. The second camera device 304 is located in the lateral viewing direction of the endoscope 30. Therefore, in the actual use of the endoscope 30, the first image plays a primary role in observing and guiding the ultrasonic endoscope 30 into place; the second image is used for the actual observation of tissues.

[0114] In an exemplary embodiment, the above-mentioned sending of encoded image data to the host computer, so that the host computer decodes the encoded image data to obtain third image data and second target image data, includes:

[0115] Encoded image data is sent to the host computer via a serial extended bus interface, so that the host computer can decode the encoded image data to obtain the third image data and the second target image data.

[0116] The serial expansion bus interface is used to enable the serial transmission of encoded image data between medical devices and host computers.

[0117] Optionally, the number of serial expansion buses in the medical device is one, or the medical device sends encoded image data to the host computer based on a serial expansion bus interface.

[0118] Alternatively, the serial expansion bus interface may include Universal Serial Bus (USB), Peripheral Component Interconnect Express (PCIE), or other interfaces.

[0119] In this embodiment, the medical device sends encoded image data to the host computer via a serial expansion bus interface, so that the host computer can decode the encoded image data to obtain third image data and second target image data. Thus, since the medical device does not need to send two image data separately to the host computer, but can send the encoded image data obtained by image encoding of the two image data to the host computer through a single serial expansion bus interface, the image display efficiency of the host computer can be significantly improved by improving the data transmission efficiency of the encoded image data.

[0120] In one exemplary embodiment, both the first storage unit and the second storage unit are on-chip storage units.

[0121] Among them, on-chip storage units refer to the storage resources located in the FPGA of medical devices.

[0122] Optionally, the on-chip storage unit may include RAM, UltraRAM, or other storage resources that can be integrated into the FPGA of the medical device.

[0123] In a straightforward manner, compared to off-chip storage resources such as Double Data Rate (DDR) that need to be located outside the FPGA, the on-chip storage units in this embodiment are located inside the FPGA. Therefore, not only can the communication rate between the two storage units and the FPGA be improved, but also the excessive communication interaction between the FPGA and the external storage units can be avoided, which would otherwise result in additional power consumption.

[0124] In this embodiment, since both the first and second storage units are on-chip storage units, and since the on-chip storage units are integrated inside the FPGA of the medical device, when the FPGA needs to read the first image data that meets the first preset condition and the data amount condition and the second image data that meets the second preset condition and the data amount condition for subsequent pixel interpolation, the FPGA in the medical device does not need to communicate with external storage resources. The FPGA only needs to read the image data from its internal on-chip storage units. Thus, while significantly improving the efficiency of image data processing, the power consumption of the medical device is significantly reduced because the medical device does not need to communicate with external storage units when reading image data.

[0125] In an exemplary embodiment, when the image size corresponding to the first image data is smaller than the image size corresponding to the second image data, the first target image data corresponds to the first image data, and the second target image data corresponds to the second image data or the second image data after pixel interpolation.

[0126] In one exemplary embodiment, the method further includes:

[0127] In response to the scaling region adjustment signal from the host computer for the second image, the adjusted second preset scaling factor is obtained; the scaling region adjustment signal is generated by the host computer when the area ratio of the target observation object in the second image is greater than or equal to the preset ratio.

[0128] When the image data with the larger corresponding image size in the first image data and the second image data is the second image data, and the data volume of the image data with the larger corresponding image size in the first image data and the second image data meets the data volume condition, pixel interpolation is performed on the image data with the larger corresponding image size in the first image data and the second image data based on the second preset scaling factor to obtain the second target image data, including:

[0129] If the amount of data in the second image data meets the data volume condition, pixel interpolation is performed on the second image data based on the adjusted second preset scaling factor to obtain the second target image data.

[0130] The target object of observation includes at least one of blood vessels, mucous membranes, or other biological tissues.

[0131] Optionally, the area proportion of the target observation object determined by the host computer in the second image can be determined by setting appropriate detection thresholds for different image features of different observation objects based on image features such as grayscale values ​​or signal strengths presented by different observation objects in the second image. This allows the pixels in the image to be classified into different categories based on the detection thresholds, thereby segmenting blood vessels, mucous membranes, or other target observation objects in the second image. Furthermore, the area proportion of the target observation object in the second image can be determined using two-dimensional image calculation methods. The area proportion is the ratio between the number of pixels of the target observation object and the number of pixels in the second image.

[0132] Optionally, deep learning algorithms can be used to annotate and train the vascular region, mucosal region, or other biological tissues in the endoscopic image to obtain an object detection model. In this way, the host computer can determine the area ratio of the target object in the second image through the object detection model.

[0133] In an exemplary embodiment, the above-mentioned response to the host computer's signal for adjusting the scaling area of ​​the second image to obtain the adjusted second preset scaling factor includes:

[0134] Based on the difference between the area ratio of the target observed object and the preset ratio, a scaling factor gain coefficient is generated.

[0135] The second preset scaling factor is adjusted based on the scaling factor gain coefficient to obtain the adjusted second preset scaling factor; the adjusted second preset scaling factor is an integer greater than or equal to the second preset scaling factor.

[0136] In an exemplary embodiment, the above-mentioned generation of scaling factor gain coefficient based on the ratio difference between the area ratio of the target observation object and the preset ratio includes:

[0137] The area proportion gain coefficient is determined based on the difference between the area proportion of the target observation object and the preset proportion.

[0138] The type gain coefficient is determined based on the type of the target observation object; if the type of the target observation object is blood vessel or mucosa, the type gain coefficient for blood vessel is smaller than that for mucosa.

[0139] Based on the area percentage gain coefficient and the type gain coefficient, a scaling factor gain coefficient is generated.

[0140] In this embodiment, when the host computer determines that the area of ​​the target object in the second image occupies a large proportion, it generates a scaling region adjustment signal and sends it to the medical device. The medical device then generates a scaling factor gain coefficient based on the difference between the area of ​​the target object and a preset proportion, as well as the type of the target object. Furthermore, it can adjust a second preset scaling factor based on this gain coefficient. Therefore, when there is a target object in the second image that medical personnel need to focus on observing, the second image can be magnified to allow for better observation of the target object. This improves the diagnostic accuracy of medical personnel by increasing the flexibility of image data processing.

[0141] In an exemplary embodiment, when the amount of data in the first target image data meets the data amount condition, the above method further includes:

[0142] Based on the first target image data, determine the pixel difference value between two rows of pixels in the first target image data;

[0143] Based on the pixel difference value between two rows of pixels in the first target image data, a third target region is determined in the first target image data; the third target region corresponds to a portion of pixels in the first target image data whose pixel difference value is greater than or equal to a preset difference value.

[0144] The first preset condition is modified based on the third target region to obtain the modified first preset condition; the modified first preset condition includes the region centered on the third target region and located within a preset distance of the third target region.

[0145] The aforementioned acquisition of first image data satisfying the first preset condition and second image data satisfying the second preset condition includes:

[0146] Collect first image data that meets the first preset condition after adjustment and second image data that meets the second preset condition.

[0147] The third target region, being a subset of pixels with significant differences in the first target image data, can be understood as the motion region within the corresponding image of the first target image data.

[0148] Optionally, the preset distance can be a distance within 5 pixels, 10 pixels, or other distances located outside any pixel at the edge of the third target region. The preset distance can be determined based on the size of the image corresponding to the first target image data.

[0149] In this embodiment, when the medical device determines that there is a moving area in the first target image data based on the first target image data, it can automatically change the first preset condition so that new first image data can be collected for the moving area. This allows medical staff to better observe the movement of the target object in the newly displayed image, thereby improving the diagnostic accuracy of medical staff by increasing the flexibility of image data processing.

[0150] The application process of the above image data processing method is illustrated below with a detailed embodiment. The image data processing method is applied to a dual-lens endoscope, which includes a first camera device, a second camera device, an FPGA, a first storage unit, a second storage unit, and a serial expansion bus interface. Assuming that the image size corresponding to the first image data is smaller than the image size corresponding to the second image data, the specific application process is as follows:

[0151] (1) Preparatory process for image data acquisition

[0152] The dual-lens endoscope receives the first and second preset conditions from the host computer via FPGA.

[0153] (2) Acquisition process of two image data

[0154] The dual-lens endoscope acquires first image data that meets the first target area through the first camera device, and acquires second image data that meets the second target area through the second camera device; the first image data corresponds to a portion of the pixels in the first image, and the second image data corresponds to a portion of the pixels in the second image.

[0155] (3) Caching process for two image data

[0156] The dual-lens endoscope uses an FPGA to cache the first image data and the second image data into the first storage unit and the second storage unit, respectively; both the first storage unit and the second storage unit are on-chip storage units.

[0157] (4) Pixel interpolation process between two image data

[0158] When the amount of data in the first image data satisfies the requirement of two rows of pixels, the dual-lens endoscope determines a 2×2 pixel matrix in the first image data through the FPGA; and performs pixel interpolation on each pixel matrix based on a first preset scaling factor to obtain the third image data.

[0159] When the image size corresponding to the third image data is larger than the image size corresponding to the second image data, and the amount of second image data in the second storage unit satisfies two rows of pixels, the dual-lens endoscope uses FPGA to perform pixel interpolation on the second image data based on the second preset scaling factor to obtain the second target image data.

[0160] (5) Encoding process of two image data

[0161] The dual-lens endoscope uses an FPGA to encode the third image data and the second target image data to obtain encoded image data.

[0162] (6) The process of transmitting encoded image data

[0163] The dual-lens endoscope sends encoded image data to the host computer via a serial extended bus interface, so that the host computer can decode the encoded image data to obtain the third image data and the second target image data.

[0164] In this embodiment, the dual-lens endoscope only caches the first image data satisfying the first target region and the second image data satisfying the second target region in the first and second storage units. Furthermore, as long as it is determined that the first image data consists of two rows of pixels in the first image and / or the second image data consists of two rows of pixels in the second image, pixel interpolation is performed on the first image based on a first preset scaling factor to obtain the third image data, and / or pixel interpolation is performed on the second image based on a second preset scaling factor to obtain the second target image data. Since image frame data caching is no longer required, and only image row data caching is needed for pixel interpolation, image data processing efficiency is greatly improved. Simultaneously, since only a portion of the image data after pixel interpolation processing using two rows of pixels from the two image data sets can be encoded, and the resulting encoded image data can be sent to the host computer, real-time image display by the host computer is achieved, resulting in high real-time display performance without image frame delay.

[0165] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0166] Based on the same inventive concept, this application also provides an image data processing apparatus for implementing the image data processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image data processing apparatus embodiments provided below can be found in the limitations of the image data processing method described above, and will not be repeated here.

[0167] In one exemplary embodiment, such as Figure 4 As shown, an image data processing device is provided, including: an acquisition module 402, a buffer module 404, an interpolation module 406, and an encoding module 408, wherein:

[0168] The acquisition module 402 is used to acquire first image data that meets a first preset condition and second image data that meets a second preset condition; the first image data corresponds to a portion of the pixels in the first image, and the second image data corresponds to a portion of the pixels in the second image.

[0169] The caching module 404 is used to cache the first image data and the second image data into the first storage unit and the second storage unit, respectively.

[0170] The interpolation module 406 is used to perform pixel interpolation on the first target image data to obtain the third image data when the data volume of the first target image data meets the data volume condition. The first target image data is the image data with the smaller corresponding image size between the first image data and the second image data.

[0171] The encoding module 408 is used to encode the third image data and the second target image data to obtain encoded image data. The second target image data is the image data other than the first target image data in the first image data and the second image data.

[0172] In an exemplary embodiment, the interpolation module 406 is specifically used to determine a 2×2 pixel matrix in the first target image data; and to perform pixel interpolation on each pixel matrix to obtain the third image data.

[0173] In one exemplary embodiment, such as Figure 4 As shown, the above-mentioned device also includes a communication module 410, which is used to send the encoded image data to the host computer so that the host computer can decode the encoded image data to obtain the third image data and the second target image data.

[0174] The communication module 410 is also used to receive the first preset condition and the second preset condition from the host computer.

[0175] In an exemplary embodiment, the interpolation module 406 is specifically used to perform pixel interpolation on the first target image data based on a first preset scaling factor to obtain the third image data.

[0176] In an exemplary embodiment, when the image size corresponding to the third image data is larger than the image size corresponding to the second target image data, the interpolation module 406 is further configured to perform pixel interpolation on the image data corresponding to the larger image size in the first image data and the second image data based on a second preset scaling factor to obtain the second target image data, provided that the amount of data of the image data corresponding to the larger image size in the first image data and the second image data meets the data amount condition.

[0177] In an exemplary embodiment, the communication module 410 is specifically used to send the encoded image data to the host computer via a serial extended bus interface, so that the host computer can decode the encoded image data to obtain the third image data and the second target image data.

[0178] Each module in the aforementioned image data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0179] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an image data processing method.

[0180] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image data processing method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0181] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0182] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0183] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0184] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image data processing method, characterized in that, The method includes: Collect first image data that meets a first preset condition and second image data that meets a second preset condition; the first image data corresponds to a portion of pixels in the first image, and the second image data corresponds to a portion of pixels in the second image; The first image data and the second image data are cached in the first storage unit and the second storage unit, respectively; If the data volume of the first target image data meets the data volume condition, pixel interpolation is performed on the first target image data to obtain the third image data. The first target image data is the image data with the smaller corresponding image size between the first image data and the second image data. The third image data and the second target image data are encoded to obtain encoded image data. The second target image data is the first image data and the image data in the second image data excluding the first target image data.

2. The method according to claim 1, characterized in that, The data volume condition is two rows of pixels.

3. The method according to claim 2, characterized in that, The step of interpolating pixels in the first target image data to obtain the third image data includes: Determine 2 in the first target image data × A 2-pixel matrix; The pixel matrix is ​​interpolated to obtain the third image data.

4. The method according to claim 1, characterized in that, The step of interpolating pixels in the first target image data to obtain the third image data includes: The first target image data is interpolated pixel by pixel based on a first preset scaling factor to obtain the third image data.

5. The method according to claim 1, characterized in that, The method further includes: The encoded image data is sent to the host computer so that the host computer can decode the encoded image data to obtain the third image data and the second target image data; The method further includes: Receive the first preset condition and the second preset condition from the host computer.

6. The method according to claim 4, characterized in that, When the image size corresponding to the third image data is larger than the image size corresponding to the second target image data, the method further includes: If the amount of data of the image data with the larger corresponding image size in the first image data and the second image data meets the data amount condition, pixel interpolation is performed on the image data with the larger corresponding image size in the first image data and the second image data based on the second preset scaling factor to obtain the second target image data.

7. The method according to claim 1, characterized in that, The first preset condition includes a first target area; The second preset condition includes the second target area.

8. The method according to claim 5, characterized in that, Sending the encoded image data to the host computer so that the host computer can decode the encoded image data to obtain the third image data and the second target image data includes: The encoded image data is sent to the host computer via a serial extended bus interface, so that the host computer can decode the encoded image data to obtain the third image data and the second target image data.

9. The method according to claim 1, characterized in that, Both the first storage unit and the second storage unit are on-chip storage units.

10. An image data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire first image data that meets a first preset condition and second image data that meets a second preset condition; the first image data corresponds to a portion of pixels in the first image, and the second image data corresponds to a portion of pixels in the second image; The caching module is used to cache the first image data and the second image data into the first storage unit and the second storage unit, respectively; The interpolation module is used to perform pixel interpolation on the first target image data to obtain the third image data when the data volume of the first target image data meets the data volume condition. The first target image data is the image data with the smaller corresponding image size between the first image data and the second image data. The encoding module is used to encode the third image data and the second target image data to obtain encoded image data, wherein the second target image data is the first image data and the image data in the second image data excluding the first target image data.

Images