Specimen image processing method, terminal device, server, and storage medium

Abstract

The embodiment of the present application discloses a specimen image processing method and a terminal device, a server, and a storage medium. The method includes: the terminal device first sends the collected initial specimen image to the server, and then obtains the specimen database corresponding to the initial specimen image from the server. When an operation instruction for the current specimen image displayed on the screen of the terminal device is detected, the terminal device can obtain specimen data matching the operation instruction according to the specimen database. The implementation of the embodiment of the present application provides an efficient interactive specimen observation method, which is conducive to improving the user's specimen observation experience.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a specimen image processing method and a terminal device, a server, and a storage medium. Background Art

[0002] In practice, it is found that users usually only observe specimens through a microscope. This method of specimen observation is single and lacks interactivity, resulting in a poor specimen observation experience for users. Summary of the invention

[0003] The embodiment of the present application discloses a specimen image processing method and a terminal device, a server, and a storage medium, providing an efficient interactive specimen observation method, which is conducive to improving the user's specimen observation experience.

[0004] The first aspect of the embodiment of the present application discloses a specimen image processing method, the method is applicable to a terminal device, the terminal device is equipped with a convex lens accessory, and the terminal device is communicatively connected with a server, the method comprising:

[0005] Using the camera of the terminal device to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image, and sending the initial specimen image to the server so that the server processes the initial specimen image to obtain a specimen database corresponding to the initial specimen image;

[0006] receiving the specimen database sent by the server; wherein the specimen database stores specimen data, the specimen data including a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image;

[0007] When an operation instruction for a current specimen image displayed on the screen of the terminal device is detected, specimen data matching the operation instruction is acquired according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image;

[0008] The sample data matching the operation instruction is displayed on the screen.

[0009] As an optional implementation, in the first aspect of the embodiment of the present application, the operation instruction includes a first amplification instruction, and acquiring specimen data matching the operation instruction according to the specimen database includes:

[0010] Obtaining a first magnification indicated by the first magnification instruction;

[0011] A first enlarged specimen image corresponding to the first magnification is obtained according to the plurality of target enlarged specimen images.

[0012] As an optional implementation, in the first aspect of the embodiment of the present application, the operation instruction includes a structure selection instruction, and the specimen data also includes pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image; the acquiring, according to the specimen database, specimen data matching the operation instruction includes:

[0013] Determining a first selected area in the current specimen image according to the structure selection instruction;

[0014] Determine, according to the first selected area, first pixel coordinate information corresponding to the first selected area in the initial specimen image;

[0015] Determining a first internal structure corresponding to the first pixel coordinate information according to pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image;

[0016] The recognition result of the first internal structure is acquired from the recognition results of each internal structure of the specimen indicated by the initial specimen image.

[0017] As an optional implementation, in the first aspect of the embodiment of the present application, determining, according to the first selected area, first pixel coordinate information corresponding to the first selected area in the initial specimen image includes:

[0018] When the current specimen image is the initial specimen image, taking the pixel coordinate information corresponding to the first selected area as the first pixel coordinate information;

[0019] When the current specimen image is an enlarged image of the initial specimen image, a first pixel coordinate mapping relationship between the current specimen image and the initial specimen image is determined according to a magnification ratio of the current specimen image relative to the initial specimen image; and according to the first pixel coordinate mapping relationship, second pixel coordinate information of the first selected area in the current specimen image is converted into first pixel coordinate information in the initial specimen image.

[0020] As an optional implementation, in the first aspect of the embodiment of the present application, the operation instruction further includes a second zoom instruction, and the method further includes:

[0021] Acquire a second magnification indicated by the second magnification instruction;

[0022] Obtaining a second enlarged specimen image corresponding to the second magnification according to the plurality of target enlarged specimen images;

[0023] According to the pixel coordinate information corresponding to the first selected area in the current specimen image and the second pixel coordinate mapping relationship between the current specimen image and the second enlarged specimen image, an enlarged image of the first internal structure is obtained from the second enlarged specimen image.

[0024] As an optional implementation, in the first aspect of the embodiment of the present application, the step of obtaining a second enlarged specimen image corresponding to the second magnification according to the multiple target enlarged specimen images includes:

[0025] When there is a second enlarged specimen image corresponding to the second magnification in the plurality of target enlarged specimen images, acquiring the second enlarged specimen image from the plurality of target enlarged specimen images;

[0026] In the case that there is no second enlarged specimen image corresponding to the second magnification among the multiple target enlarged specimen images, an enlarged specimen image with a magnification smaller than the second magnification and closest to the second magnification is used as a target specimen image among the multiple target enlarged specimen images; and the target specimen image is enlarged by using an interpolation algorithm to obtain a second enlarged specimen image corresponding to the second magnification.

[0027] As an optional implementation, in the first aspect of the embodiment of the present application, using the camera of the terminal device to photograph the slide placed on the convex lens accessory to obtain an initial specimen image includes:

[0028] Using the camera of the terminal device, continuously photographing the glass slide placed on the convex lens accessory within a preset time period to obtain multiple frames of specimen images;

[0029] Calculate the image difference value between each two adjacent frames in the multiple frames of specimen images by using a hash algorithm;

[0030] If the image difference values ​​of each two adjacent frames in the multiple frames of specimen images are all less than the image difference threshold, it is determined that the glass slide is in a stationary state;

[0031] A specimen image of any frame among the plurality of frames of specimen images is used as an initial specimen image.

[0032] As an optional implementation, in the first aspect of the embodiment of the present application, the multiple target enlarged specimen images corresponding to the initial specimen image are obtained by the server processing the initial specimen image using a super-resolution reconstruction algorithm; the recognition results of the various internal structures of the specimen indicated by the initial specimen image are obtained by the server processing the initial specimen image using an instance segmentation algorithm;

[0033] The specimen data also includes specimen category information of the specimen indicated by the initial specimen image.

[0034] A second aspect of an embodiment of the present application discloses a specimen image processing method, the method being applicable to a server, the server being connected to a terminal device, the terminal device being equipped with a convex lens accessory, comprising:

[0035] receiving an initial specimen image sent by the terminal device, wherein the initial specimen image is an image obtained by the terminal device photographing a glass slide placed on the convex lens accessory through a camera;

[0036] Processing the initial specimen image to obtain specimen data corresponding to the initial specimen image, and constructing a specimen database corresponding to the initial specimen image based on the specimen data, wherein the specimen data includes a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image;

[0037] The specimen database is sent to the terminal device so that when the terminal device detects an operation instruction for a current specimen image displayed on a screen of the terminal device, the terminal device acquires specimen data matching the operation instruction according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image.

[0038] As an optional implementation, in the second aspect of the embodiment of the present application, the specimen data further includes pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image; the processing of the initial specimen image to obtain the specimen data corresponding to the initial specimen image includes:

[0039] The initial specimen image is magnified according to a preset magnification to obtain a plurality of initial magnified specimen images corresponding to the initial specimen image, and each initial magnified specimen image is processed by a super-resolution reconstruction algorithm to obtain a plurality of target magnified specimen images corresponding to the initial specimen image;

[0040] The initial specimen image is processed by using an instance segmentation algorithm to obtain recognition results and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image.

[0041] As an optional implementation, in the second aspect of the embodiment of the present application, the first initial enlarged specimen image is any one of the multiple initial enlarged specimen images, and the use of a super-resolution reconstruction algorithm to process each initial enlarged specimen image to obtain multiple target enlarged specimen images corresponding to the initial specimen image includes:

[0042] Extracting a first image feature of the first initial enlarged specimen image through a pre-trained image super-resolution reconstruction network, and performing residual processing on the first image feature to obtain a residual processing result;

[0043] Performing upsampling processing on the first image feature to obtain a first upsampling result, and performing upsampling processing on the residual processing result to obtain a second upsampling processing result;

[0044] An upsampling fusion result is obtained according to the first upsampling result and the second upsampling processing result, and super-resolution reconstruction is performed on the upsampling fusion result to obtain a target enlarged specimen image corresponding to the initial specimen image.

[0045] As an optional implementation, in the second aspect of the embodiment of the present application, the initial specimen image is processed using an instance segmentation algorithm to obtain the recognition result and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image, including:

[0046] Extracting a structural feature image and a pixel feature image of the initial specimen image using a pre-trained image instance segmentation model;

[0047] In the image instance segmentation model, determining, according to the structural feature image, a recognition result of each internal structure of the specimen indicated by the initial specimen image and a first position of each internal structure on the initial specimen image;

[0048] In the image instance segmentation model, a background mask image, a foreground mask image and a transition zone mask image between the background and the foreground of each internal structure corresponding to each internal structure are generated according to the pixel feature image, and a structure segmentation mask image corresponding to each internal structure is generated according to the pixel feature image, the background mask image, the foreground mask image and the transition zone mask image;

[0049] The structure segmentation mask image corresponding to each internal structure is used to correct the first position of each internal structure on the initial specimen image to obtain pixel coordinate information of each internal structure.

[0050] As an optional implementation, in the second aspect of the embodiment of the present application, after obtaining the pixel coordinate information and recognition result of each internal structure of the specimen indicated by the initial specimen image according to the category and position of each internal structure and the structure segmentation mask image, the method further includes:

[0051] According to the recognition results of each internal structure of the specimen indicated by the initial specimen image, the specimen category corresponding to the initial specimen image is determined from the specimen category library; wherein the specimen category library includes multiple specimen categories and the recognition results of each internal structure of each specimen category.

[0052] A third aspect of an embodiment of the present application discloses a terminal device, wherein the terminal device is equipped with a convex lens accessory and is communicatively connected with a server, including:

[0053] A collection unit, used to use the camera of the terminal device to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image;

[0054] A first transceiver unit is used to send the initial specimen image to the server so that the server processes the initial specimen image to obtain a specimen database corresponding to the initial specimen image; and receive the specimen database sent by the server; wherein the specimen database stores specimen data, the specimen data including a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image;

[0055] A first processing unit is configured to, when an operation instruction for a current specimen image displayed on a screen of the terminal device is detected, acquire specimen data matching the operation instruction according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image;

[0056] The display unit is used to display the specimen data matching the operation instruction on the screen.

[0057] A fourth aspect of an embodiment of the present application discloses a terminal device, including:

[0058] A memory storing executable program code;

[0059] a processor coupled to the memory;

[0060] The processor calls the executable program code stored in the memory to execute any one of the methods of the first aspect of the present application.

[0061] A fifth aspect of an embodiment of the present application discloses a server, wherein the server is connected to a terminal device, and the terminal device is equipped with a convex lens accessory, including:

[0062] A second transceiver unit is used to receive an initial specimen image sent by the terminal device, wherein the initial specimen image is an image obtained by the terminal device photographing the glass slide placed on the convex lens accessory through a camera;

[0063] a second processing unit, configured to process the initial specimen image to obtain specimen data corresponding to the initial specimen image, and to construct a specimen database corresponding to the initial specimen image according to the specimen data, wherein the specimen data includes a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image;

[0064] The second transceiver unit is also used to send the specimen database to the terminal device, so that when the terminal device detects an operation instruction for the current specimen image displayed on the screen of the terminal device, the terminal device obtains specimen data matching the operation instruction according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image.

[0065] A sixth aspect of an embodiment of the present application discloses a server, including:

[0066] A memory storing executable program code;

[0067] a processor coupled to the memory;

[0068] The processor calls the executable program code stored in the memory to execute any one of the methods of the second aspect of the present application.

[0069] A seventh aspect of an embodiment of the present application discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any one of the methods of the first aspect or the second aspect of the present application.

[0070] The eighth embodiment of the present application discloses a computer program product. When the computer program product runs on a computer, the computer executes any one of the methods of the first aspect or the second aspect.

[0071] A ninth aspect of an embodiment of the present application discloses an application publishing system, which is used to publish a computer program product. When the computer program product runs on a computer, the computer executes any one of the methods of the first aspect or the second aspect.

[0072] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0073] In implementing the embodiment of the present application, the terminal device first sends the collected initial specimen image to the server, and then obtains the specimen database corresponding to the initial specimen image from the server. When detecting an operation instruction for the current specimen image displayed on the screen of the terminal device, the terminal device can obtain specimen data matching the operation instruction according to the specimen database. Since the specimen data in the specimen database includes multiple target enlarged specimen images corresponding to the initial specimen image and the recognition results of each internal structure of the specimen indicated by the initial specimen image, when the terminal device detects the operation instruction, it can quickly obtain the specimen data matching the operation instruction from the specimen database, and quickly display the specimen data matching the operation instruction on the display screen of the terminal device, providing an efficient interactive specimen observation method, which is conducive to improving the user's specimen observation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0075] Figure 1 It is a schematic diagram of an application scenario of an image capturing method disclosed in an embodiment of the present application;

[0076] Figure 2A It is a schematic diagram of a terminal device equipped with a convex lens accessory disclosed in an embodiment of the present application;

[0077] Figure 2B is a schematic diagram of a terminal device disclosed in an embodiment of the present application without a convex lens accessory;

[0078] Figure 3 It is a flow chart of a specimen image processing method disclosed in an embodiment of the present application;

[0079] Figure 4A It is a schematic diagram of an interface corresponding to the zoom-in instruction disclosed in the embodiment of the present application;

[0080] Figure 4B It is a schematic diagram of an interface corresponding to the structure selection instruction disclosed in the embodiment of the present application;

[0081] Figure 5 It is a flowchart of another specimen image processing method disclosed in an embodiment of the present application;

[0082] Figure 6It is a schematic diagram of a process of obtaining a target enlarged specimen image corresponding to a first initial enlarged specimen image disclosed in an embodiment of the present application;

[0083] Figure 7 It is a schematic diagram of a process for obtaining recognition results and pixel coordinate information of each internal structure of a specimen indicated by an initial specimen image disclosed in an embodiment of the present application;

[0084] Figure 8 It is a structural block diagram of a terminal device disclosed in an embodiment of the present application;

[0085] Fig. 9 It is a structural block diagram of a server disclosed in an embodiment of the present application;

[0086] Fig.10 It is a structural block diagram of a terminal device disclosed in an embodiment of the present application;

[0087] Fig.11 It is a structural block diagram of a server disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0089] It should be noted that the terms "including", "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0090] The specimen image processing method, terminal device, and server disclosed in the embodiments of the present application can ensure the real-time observation of users.

[0091] See also Figure 1 , Figure 1 Schematic diagram of a scene of a specimen image processing method disclosed in an embodiment of the present application. Figure 1As shown, it includes a terminal device 100, a glass slide 200 and a server 300. The terminal device 100 uses its camera to shoot the glass slide 200 to obtain an initial specimen image, and sends the initial specimen image to the server 300, so that the server 300 can obtain a specimen database corresponding to the initial specimen image by processing the initial specimen image. The specimen data of the specimen database includes multiple target enlarged specimen images corresponding to the initial specimen image, and the recognition results of each internal structure of the specimen indicated by the initial specimen image. The server 300 sends the specimen database corresponding to the initial specimen image to the terminal device 100, so that when the terminal device 100 detects an operation instruction for the current specimen image displayed on the screen of the terminal device 100, it can obtain the specimen data matching the operation instruction according to the specimen database. Based on this method, the user can quickly obtain the relevant knowledge of the enlarged specimen image or any internal structure of the specimen indicated by the initial specimen image by operating the specimen image displayed on the display screen of the terminal device.

[0092] To obtain the above-mentioned initial specimen image, in some embodiments, as Figure 2A as well as Figure 2B As shown, the terminal device 100 may include a camera module 110 and a light source module 120. In addition, the terminal device 100 may be detachably equipped with a convex lens accessory 130. Figure 2A FIG. 1 shows a schematic diagram of a terminal device 100 when the convex lens accessory 130 is assembled. Figure 2B The schematic diagram shows the terminal device 100 when the convex lens accessory 130 is not installed. When the convex lens accessory 130 is installed on the terminal device 100, the convex lens accessory 130 can magnify the shooting target within its framing range, so that the camera module 110 can capture the magnified image through the convex lens accessory 130, and the light source module 120 can be used to provide sufficient light for the above-mentioned image collection process to avoid the image collected by the camera module 110 being too dark, blurred, etc. due to insufficient light.

[0093] It should be noted that Figure 2A as well as Figure 2B The convex lens accessory 130 shown is mounted on the back of the terminal device 100 to cooperate with the rear camera module 110 of the terminal device 100. This is only an example and does not constitute a limitation on the mounting position of the convex lens accessory 130 in the embodiment of the present application. In some embodiments, the convex lens accessory 130 can also be mounted on the front of the terminal device 100 to cooperate with the front camera module 110 of the terminal device 100 to achieve the shooting of objects on a microscopic scale.

[0094] Exemplarily, the terminal device 100 in the embodiment of the present application may include various devices or systems with a camera module 110 and a light source module 120, such as a mobile phone, a tablet computer, a smart wearable device, a PC (Personal Computer), a PDA (Personal Digital Assistant), etc. Among them, the light source module 120 can reuse the flash on the terminal device 100, or it can use other light sources to be independently set. Exemplarily, the convex lens 133 in the above-mentioned convex lens accessory 130 can include various lenses with a certain magnification (such as 100 times, 200 times, etc.), and can cooperate with the adjustment of the imaging position to achieve an adjustable magnification.

[0095] The embodiment of the present application discloses a specimen image processing method and a terminal device, a server, and a storage medium, providing an efficient interactive specimen observation method, which is conducive to improving the user's specimen observation experience.

[0096] See also Figure 3 , Figure 3 is a flow chart of a specimen image processing method disclosed in an embodiment of the present application, Figure 3 The specimen image processing method shown is applied to the above-mentioned terminal device 100. It includes:

[0097] 401. The terminal device uses its camera to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image.

[0098] In an embodiment of the present application, the initial specimen image captured by the camera module of the terminal device is an image obtained by the camera module capturing the glass slide through a convex lens accessory. It is understandable that the convex lens accessory can first physically enlarge the glass slide, and then the camera module captures the physically enlarged image to obtain the initial specimen image. The multiple of physical magnification can be obtained by multiplying the objective lens magnification of the convex lens accessory by the eyepiece magnification. Exemplarily, the initial specimen image can be the image of the specimen when it is physically magnified 35 times. It is understandable that the initial specimen image can be observed by the user with the naked eye.

[0099] In some embodiments, the terminal device uses its camera to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image, which may include: the terminal device uses its camera to continuously photograph the glass slide placed on the convex lens accessory within a preset time length to obtain multiple frames of specimen images; uses a hash algorithm to calculate the image difference value of each two adjacent frames in the multiple frames of specimen images; if the image difference value of each two adjacent frames in the multiple frames of specimen images is less than the image difference threshold, it is determined that the glass slide is in a stationary state, and the specimen image of any frame in the multiple frames of specimen images is used as the initial specimen image.

[0100] Furthermore, the above hash algorithm may be a mean hash algorithm, a difference hash algorithm or a perceptual hash algorithm, which is not limited here.

[0101] In some embodiments, if the above-mentioned hash algorithm is a mean hash algorithm, taking the adjacent first specimen image and the second specimen image in the multiple frames of specimen images captured by the camera as an example, using the hash algorithm to calculate the image difference value of each adjacent two frames in the multiple frames of specimen images, it can include: reducing the size of the first specimen image and the second specimen image to obtain a first specimen image of a preset size and a second specimen image of a preset size; performing grayscale conversion on the first specimen image of the preset size to obtain a first grayscale matrix matching the preset size, and performing grayscale conversion on the second specimen image of the preset size to obtain a second grayscale matrix matching the preset size; obtaining a first average grayscale value according to the first grayscale matrix, and obtaining a second average grayscale value according to the second grayscale matrix; obtaining first fingerprint information of the first specimen image according to the first grayscale matrix and the first average grayscale value; obtaining second fingerprint information of the second specimen image according to the second grayscale matrix and the second average grayscale value; obtaining the image difference value between the first specimen image and the second specimen image according to the first fingerprint information and the second fingerprint information.

[0102] The following is an example of the first fingerprint information:

[0103] The first grayscale matrix may include the grayscale value of each pixel included in the first specimen image of a preset size. The terminal device obtains the first fingerprint information of the first specimen image according to the first grayscale matrix and the first average grayscale value, which may include: if the grayscale value of any pixel is greater than or equal to the first average grayscale value, then the pixel is determined to correspond to the first fingerprint data; if the grayscale value of any pixel is less than the first average grayscale value, then the pixel is determined to correspond to the second fingerprint data.

[0104] The fingerprint data (first fingerprint data or second fingerprint data) corresponding to each pixel included in the first specimen image of the preset size is combined to obtain the first fingerprint information.

[0105] Exemplarily, if the preset size is 8*8, the first / second grayscale matrix is ​​8*8, and the first / second fingerprint information is 64-bit data.

[0106] In some embodiments, obtaining an image difference value between the first specimen image and the second specimen image according to the first fingerprint information and the second fingerprint information includes: calculating a Hamming distance between the first fingerprint information and the second fingerprint information, and obtaining an image difference value between the first specimen image and the second specimen image according to the Hamming distance.

[0107] It should be noted that the Hamming distance represents the different number of bits in the first fingerprint information and the second fingerprint information. The larger the Hamming distance, the greater the image difference between the first specimen image and the second specimen image. Conversely, the smaller the image difference between the first specimen image and the second specimen image.

[0108] In some embodiments, the terminal device can preset multiple Hamming distance values, each Hamming distance value corresponds to a first image difference value, and the image difference values ​​corresponding to different Hamming distance values ​​are different. In this way, the terminal device can obtain the image difference value between the first specimen image and the second specimen image based on the calculated Hamming distance.

[0109] By implementing this method, when it is detected that the slide is in a static state, the terminal device will use the collected specimen image as the initial specimen image and send the initial specimen image to the server. This can ensure the validity of the initial specimen image sent by the terminal device to the server and further improve the efficiency of interactive specimen observation.

[0110] 402. The terminal device sends an initial specimen image to the server.

[0111] In some embodiments, the terminal device may also detect the current network rate of the terminal device before executing step 402, and send the initial specimen image to the server when the current network rate of the terminal device is greater than or equal to the network rate threshold. By implementing this method, the data interaction between the terminal device and the server can be carried out under good network conditions, which is conducive to ensuring the efficiency of data interaction.

[0112] Correspondingly, the server receives the initial specimen image sent by the terminal device.

[0113] 403. The server processes the initial specimen image to obtain specimen data corresponding to the initial specimen image, and constructs a specimen database corresponding to the initial specimen image based on the specimen data, wherein the specimen data includes a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image.

[0114] For example, the multiple target magnified specimen images corresponding to the initial specimen image may include specimen images magnified 2 times, 3 times, 4 times, and 5 times, etc. corresponding to the initial specimen image. The recognition results of each internal structure of the specimen indicated by the initial specimen image may include the name and function of the structure, etc.

[0115] 404. The server sends the specimen database to the terminal device.

[0116] Correspondingly, the terminal device receives the specimen database sent by the server.

[0117] 405. When the terminal device detects an operation instruction for the current specimen image displayed on its screen, the terminal device acquires specimen data matching the operation instruction according to the specimen database, and displays the specimen data matching the operation instruction on the screen. The current specimen image is an initial specimen image or an enlarged image of the initial specimen image.

[0118] In some embodiments, the above operation instructions may be triggered by the user through voice or gestures, wherein the gestures may include but are not limited to pressing, sliding, etc. Optionally, the gesture triggering method may include contact gesture triggering and non-contact gesture triggering.

[0119] In some embodiments, the above-mentioned operation instruction can be divided into two types, one is a magnification instruction for magnifying the current specimen image, and the other is a structure selection instruction for selecting any structural area in the current specimen image. If the above-mentioned operation instruction is a magnification instruction, the specimen data matched with the operation instruction is the magnified image corresponding to the current specimen image, and if the above-mentioned operation instruction is a structure selection instruction, the specimen data matched with the operation instruction is the recognition result of a certain internal structure of the specimen.

[0120] For example, see Figure 4A , Figure 4A This is a schematic diagram of the interface corresponding to the zoom command; please refer to Figure 4B , Figure 4B It is a schematic diagram of the interface corresponding to the structure selection instruction.

[0121] By implementing the above method, since the specimen data in the specimen database includes multiple target enlarged specimen images corresponding to the initial specimen image and the recognition results of various internal structures of the specimen indicated by the initial specimen image, when the terminal device detects an operation instruction, it can quickly obtain the specimen data matching the operation instruction from the specimen database and quickly display the specimen data matching the operation instruction on the display screen of the terminal device, thereby providing an efficient and interactive specimen observation method, which is conducive to improving the user's specimen observation experience.

[0122] See also Figure 5 , Figure 5 This is a flow chart of another specimen image processing method disclosed in the embodiment of the present application. It includes:

[0123] 501. The terminal device uses its camera to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image.

[0124] 502. The terminal device sends an initial specimen image to the server.

[0125] For the description of steps 501-502, please refer to Figure 3The introduction of step 401 to step 402 in the illustrated embodiment will not be repeated here.

[0126] In the embodiment of the present application, the specimen data in the specimen database may also include pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image.

[0127] 503. The server magnifies the initial specimen image according to a preset magnification to obtain a plurality of initial magnified specimen images corresponding to the initial specimen image, and processes each initial magnified specimen image using a super-resolution reconstruction algorithm to obtain a plurality of target magnified specimen images corresponding to the initial specimen image.

[0128] 504. The server processes the initial specimen image using an instance segmentation algorithm to obtain recognition results and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image.

[0129] 505. The server obtains a specimen database corresponding to the initial specimen image according to the multiple target enlarged specimen images corresponding to the initial specimen image and the recognition results and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image.

[0130] 506. The server sends the specimen database to the terminal device.

[0131] 507. When the terminal device detects a first zoom-in instruction for the current specimen image displayed on its screen, it obtains a first magnification indicated by the first zoom-in instruction, and amplifies the specimen image according to multiple targets corresponding to the initial specimen image to obtain a first magnified specimen image corresponding to the first magnification, and displays the first magnified specimen image on the screen.

[0132] The current specimen image is the initial specimen image or an enlarged image of the initial specimen image.

[0133] 508. When the terminal device detects a structure selection instruction for the current specimen image displayed on its screen, the terminal device determines a first selection area in the current specimen image according to the structure selection instruction, determines first pixel coordinate information corresponding to the first selection area in the initial specimen image according to the first selection area, determines a first internal structure corresponding to the first pixel coordinate information according to the pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image, obtains a recognition result of the first internal structure from the recognition results of each internal structure of the specimen indicated by the initial specimen image, and displays the recognition result of the first internal structure on the screen.

[0134] In some embodiments, the terminal device determines, based on the first selected area, first pixel coordinate information corresponding to the first selected area in the initial specimen image, which may include: when the current specimen image is the initial specimen image, using the pixel coordinate information corresponding to the first selected area as the first pixel coordinate information; when the current specimen image is an enlarged image of the initial specimen image, determining the first pixel coordinate mapping relationship between the current specimen image and the initial specimen image based on the magnification of the current specimen image relative to the initial specimen image; and according to the first pixel coordinate mapping relationship, converting the second pixel coordinate information of the first selected area in the current specimen image into the first pixel coordinate information in the initial specimen image.

[0135] In some embodiments, the above-mentioned operation instruction may also include a second zoom instruction. After determining the first selected area in the current specimen image according to the structure selection instruction, the terminal device may also obtain the second magnification indicated by the second zoom instruction; according to the multiple target enlarged specimen images corresponding to the initial specimen image, obtain the second enlarged specimen image corresponding to the second magnification; according to the pixel coordinate information corresponding to the first selected area in the current specimen image, and the second pixel coordinate mapping relationship between the current specimen image and the second enlarged specimen image, obtain the enlarged image of the first internal structure from the second enlarged specimen image. Furthermore, the enlarged image of the first internal structure may also be displayed on the display screen of the terminal device. By implementing this method, when any internal structure of the current specimen image is identified, the internal structure may be enlarged. It should be noted that the identification and enlargement of any internal structure may be performed synchronously or asynchronously, which is not limited here.

[0136] In some embodiments, the terminal device acquires the second enlarged specimen image / the first enlarged specimen image, which may include: when the second enlarged specimen image / the second enlarged specimen image exists in a plurality of target enlarged specimen images corresponding to the initial specimen image, acquiring the second enlarged specimen image / the second enlarged specimen image from the plurality of target enlarged specimen images; when the second enlarged specimen image / the first enlarged specimen image does not exist in the plurality of target enlarged specimen images, taking, among the plurality of target enlarged specimen images, an enlarged specimen image having a magnification less than the second magnification / the first magnification and closest to the second magnification / the first magnification as a target specimen image; and enlarging the target specimen image using an interpolation algorithm to obtain a second enlarged specimen image corresponding to the second magnification.

[0137] Exemplarily, the multiple target magnified specimen images corresponding to the initial specimen image include magnified specimen images of the initial specimen image at 2 times, 2.5 times, 3 times, 3.5 times, 4 times, 4.5 times, 5 times, 5.5 times, 6 times, and 6.5 times. When the current specimen image is the initial specimen image, the first magnification / second magnification is 3.4, and the target specimen image is an enlarged specimen image 3 times of the initial specimen image.

[0138] In some embodiments, the above interpolation algorithm may include nearest neighbor difference, bilinear interpolation or bicubic interpolation, etc., which is not limited here.

[0139] In the embodiment of the present application, the first initial magnified specimen image is any one of a plurality of initial magnified specimen images. Figure 6 , Figure 6 This is a schematic diagram of a process for obtaining a target enlarged specimen image corresponding to a first initial enlarged specimen image disclosed in an embodiment of the present application. It may include:

[0140] 610. The server extracts a first image feature of the first initial enlarged specimen image through a pre-trained image super-resolution reconstruction network.

[0141] In some embodiments, the image super-resolution reconstruction network may include a feature extraction module, which may include an input unit and a convolution unit. The server may first input the first initial enlarged specimen image through the input unit, and then process the first initial enlarged specimen image through the convolution unit to obtain the first image feature.

[0142] 620. The server performs residual processing on the first image feature to obtain a residual processing result.

[0143] In some embodiments, the image super-resolution reconstruction network may include multiple residual units, and further, the residual unit may be a dense residual unit in a weight adaptive residual. The server may use the multiple residual units to process the first image feature to obtain a residual processing result.

[0144] Exemplarily, the plurality of residual units include residual unit 1, residual unit 2, and residual unit 3. The server may first process the first image feature through residual unit 1 to obtain a first processing result, then process the first processing result through residual unit 2 to obtain a second processing result, and then process the second processing result through residual unit 3 to obtain the residual processing result. By implementing this method, the first image feature is subjected to residual processing using a plurality of residual units, and the accuracy of obtaining the residual processing result can be improved.

[0145] 630. The server performs upsampling processing on the first image feature to obtain a first upsampling result, performs upsampling processing on the residual processing result to obtain a second upsampling processing result, and obtains an upsampling fusion result based on the first upsampling processing result and the second upsampling processing result.

[0146] In some embodiments, the image super-resolution reconstruction network may include N upsampling units, where N may be a positive even number, such as 2, 4, 6, 8, etc. The server may perform upsampling processing on the first image feature through N / 2 upsampling units to obtain a first upsampling result, and perform upsampling processing on the residual processing result through N / 2 upsampling units to obtain a second upsampling result.

[0147] Exemplarily, if N=4, the four upsampling units may include upsampling unit 1 , upsampling unit 2 , upsampling unit 3 , and upsampling unit 4 .

[0148] The server upsamples the first image feature through N / 2 upsampling units to obtain a first upsampling result, which may include: the server first upsamples the first image feature through upsampling unit 1 to obtain a first output result, and then upsamples the first output result through upsampling unit 3 to obtain a first upsampling result.

[0149] The server performs upsampling processing on the residual processing result through N / 2 upsampling units to obtain a second upsampling result, which may include: the server first performs upsampling processing on the residual processing result through upsampling unit 2 to obtain a second output result, and splices the first output result and the second output result to obtain a spliced ​​output result, and finally performs upsampling processing on the spliced ​​output result through upsampling unit 4 to obtain a second upsampling result.

[0150] In the embodiment of the present application, the upsampling fusion result can be obtained by splicing the first upsampling result and the second upsampling result.

[0151] 640. The server performs super-resolution reconstruction on the up-sampling fusion result to obtain a target enlarged specimen image corresponding to the initial specimen image.

[0152] The resolution of the first initial magnified specimen image is smaller than the resolution of the target magnified specimen image.

[0153] In some embodiments, the connection mode between the multiple residual units and the multiple upsampling units can adopt the connection mode of the dense residual network, which can make the network converge faster on the one hand, and reduce the generation of artifacts on the other hand, thereby improving the accuracy of image reconstruction.

[0154] In some embodiments, see Figure 7 , Figure 7 This is a flow chart of obtaining the recognition results and pixel coordinate information of each internal structure of a specimen indicated by an initial specimen image disclosed in an embodiment of the present application. It includes:

[0155] 710. The server extracts a structural feature image and a pixel feature image of the initial specimen image using a pre-trained image instance segmentation model.

[0156] 720. In the image instance segmentation model, the server determines, based on the structural feature image, a recognition result of each internal structure of the specimen indicated by the initial specimen image and a first position of each internal structure on the initial specimen image.

[0157] The image embodiment segmentation model may include a convolutional neural network for identifying each internal structure of the specimen indicated by the initial specimen image. The server may determine the identification result of each internal structure of the specimen indicated by the initial specimen image and the first position of each internal structure on the initial specimen image through the convolutional neural network.

[0158] In the embodiment of the present application, the first position of each internal structure on the initial specimen image can be represented by initial pixel coordinate information corresponding to the internal structure, and the initial pixel coordinate information corresponding to the internal structure may include the initial coordinates of each pixel corresponding to the internal structure.

[0159] 730. In the image instance segmentation model, the server generates a background mask image, a foreground mask image, and a transition band mask image between the background and foreground of each internal structure corresponding to each internal structure based on the pixel feature image, and generates a structure segmentation mask image corresponding to each internal structure based on the pixel feature image, the background mask image, the foreground mask image, and the transition band mask image.

[0160] Among them, the background mask image can be an image of background pixels of internal structure represented by pixels with pixel value "1", the foreground mask image can be an image of foreground pixels of internal structure represented by pixels with pixel value "0", and the transition zone mask image can be an image of transition pixels of internal structure represented by pixels with pixel values ​​between "0" and "1".

[0161] 740. The server uses the structure segmentation mask image of each internal structure to correct the first position of each internal structure on the initial specimen image, and obtains pixel coordinate information of each internal structure.

[0162] By implementing this method, when performing example segmentation on the initial specimen image, not only the foreground mask image and the background mask image of the image are focused on, but also the transition mask image between the foreground and the background is focused on, which greatly improves the accuracy of the segmentation and further improves the accuracy of the pixel coordinate information of each internal structure.

[0163] In some embodiments, the specimen database may further include a specimen category corresponding to the initial specimen image. After step 740, the server may further determine the category of the specimen indicated by the initial specimen image from the specimen category library based on the recognition results of the various internal structures of the specimen indicated by the initial specimen image.

[0164] Optionally, a specimen category library may be preset in the server, which may store a plurality of specimen categories and identification results of various internal structures corresponding to each specimen category. The server may determine the category of the specimen indicated by the initial specimen image from the specimen category library based on the identification results of various internal structures of the specimen indicated by the initial specimen image.

[0165] By implementing the above method, multiple target enlarged specimen images corresponding to the initial specimen image in the specimen database are constructed by the server based on the image super-resolution reconstruction network, which ensures the resolution of the multiple target enlarged specimen images corresponding to the initial specimen image and improves the clarity of the specimen image observed by the user. The server obtains the recognition results and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image based on the instance segmentation algorithm, which greatly improves the accuracy of the recognition results and pixel coordinate information of each internal structure, thereby improving the user's observation experience.

[0166] See also Figure 8 , Figure 8 This is a block diagram of a terminal device disclosed in an embodiment of the present application. It includes: a collection unit 801, a transceiver unit 802, an acquisition unit 803 and a display unit 804, wherein:

[0167] The acquisition unit 801 is used to use the camera of the terminal device to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image;

[0168] The first transceiver unit 802 is used to send an initial specimen image to a server so that the server processes the initial specimen image to obtain a specimen database corresponding to the initial specimen image; and receive the specimen database sent by the server; wherein the specimen database stores specimen data, and the specimen data includes a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image;

[0169] The first processing unit 803 is used to acquire specimen data matching the operation instruction according to the specimen database when an operation instruction for the current specimen image displayed on the screen of the terminal device is detected; wherein the current specimen image is an initial specimen image or an enlarged image of the initial specimen image;

[0170] The display unit 804 is used to display the specimen data matching the operation instruction on the screen.

[0171] In some embodiments, the operation instruction may include a first magnification instruction, and the first processing unit 803 is specifically used to obtain a first magnification indicated by the first magnification instruction; and to obtain a first magnified specimen image corresponding to the first magnification according to the above-mentioned multiple target magnification images.

[0172] In some embodiments, the operation instruction may include a structure selection instruction, and the specimen data also includes pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image; the first processing unit 803 is specifically used to determine a first selection area in the current specimen image according to the structure selection instruction; determine the first pixel coordinate information corresponding to the first selection area in the initial specimen image according to the first selection area; determine the first internal structure corresponding to the first pixel coordinate information according to the pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image; and obtain the recognition result of the first internal structure from the recognition results of the internal structures of the specimen indicated by the initial specimen image.

[0173] In some embodiments, the first processing unit 803 is used to determine the first pixel coordinate information corresponding to the first selected area in the initial specimen image according to the first selected area, which may specifically include:

[0174] The first processing unit 803 is used to use the pixel coordinate information corresponding to the first selected area as the first pixel coordinate information when the current specimen image is the initial specimen image; and when the current specimen image is an enlarged image of the initial specimen image, determine the first pixel coordinate mapping relationship between the current specimen image and the initial specimen image according to the magnification of the current specimen image relative to the initial specimen image; and convert the second pixel coordinate information of the first selected area in the current specimen image into the first pixel coordinate information in the initial specimen image according to the first pixel coordinate mapping relationship.

[0175] In some embodiments, the above-mentioned operation instruction also includes a second zoom-in instruction, and the first processing unit 803 is further used to obtain a second magnification indicated by the second zoom-in instruction after determining the first selected area in the current specimen image according to the structure selection instruction; obtain a second magnification corresponding to the second magnification by zooming in the specimen image according to the above-mentioned multiple targets; and obtain an enlarged image of the first internal structure from the second enlarged specimen image according to the pixel coordinate information corresponding to the first selected area in the current specimen image and the second pixel coordinate mapping relationship between the current specimen image and the second enlarged specimen image.

[0176] In some embodiments, the first processing unit 803 is used to magnify the specimen image according to the above-mentioned multiple targets to obtain the second magnified specimen image corresponding to the second magnification, which may specifically include:

[0177] The first processing unit 803 is used for obtaining the second enlarged specimen image from the multiple target enlarged specimen images when there is a second enlarged specimen image corresponding to the second magnification among the multiple target enlarged specimen images; and, when there is no second enlarged specimen image corresponding to the second magnification among the multiple target enlarged specimen images, taking an enlarged specimen image with a magnification smaller than the second magnification and closest to the second magnification as the target specimen image among the multiple target enlarged specimen images; and enlarging the target specimen image by using an interpolation algorithm to obtain the second enlarged specimen image corresponding to the second magnification.

[0178] In some embodiments, the acquisition unit 801 is specifically used to use the camera of the terminal device to continuously shoot the slide placed on the convex lens accessory within a preset time length to obtain multiple frames of specimen images; use a hash algorithm to calculate the image difference value of each two adjacent frames in the multiple frames of specimen images; if the image difference value of each two adjacent frames in the multiple frames of specimen images is less than the image difference threshold, it is determined that the slide is in a stationary state; and use the specimen image of any frame in the multiple frames of specimen images as the initial specimen image.

[0179] In some embodiments, multiple target enlarged specimen images corresponding to the initial specimen image are obtained by the server processing the initial specimen image using a super-resolution reconstruction algorithm; the recognition results of various internal structures of the specimen indicated by the initial specimen image are obtained by the server processing the initial specimen image using an instance segmentation algorithm; the specimen data also includes specimen category information of the specimen indicated by the initial specimen image.

[0180] See also Fig. 9 , Fig. 9 It is a structural block diagram of a server disclosed in an embodiment of the present application. Fig. 9 The server shown is connected to a terminal device, and the terminal device is equipped with a convex lens accessory, including a second transceiver unit 901 and a second processing unit 902; wherein:

[0181] The second transceiver unit 901 is used to receive an initial specimen image sent by a terminal device, where the initial specimen image is an image obtained by the terminal device taking a picture of a glass slide placed on the convex lens accessory through a camera;

[0182] The second processing unit 902 is used to process the initial specimen image to obtain specimen data corresponding to the initial specimen image, and to construct a specimen database corresponding to the initial specimen image according to the specimen data, wherein the specimen data includes a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image;

[0183] The second transceiver unit 901 is also used to send the above-mentioned specimen database to the terminal device, so that when the terminal device detects an operation instruction for the current specimen image displayed on the screen of the terminal device, it obtains specimen data matching the operation instruction according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image.

[0184] In some embodiments, the specimen data may also include pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image; the second processing unit 902 is specifically used to magnify the initial specimen image according to a preset magnification to obtain multiple initial magnified specimen images corresponding to the initial specimen image, and use a super-resolution reconstruction algorithm to process each initial magnified specimen image to obtain multiple target magnified specimen images corresponding to the initial specimen image; and use an instance segmentation algorithm to process the initial specimen image to obtain recognition results and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image.

[0185] In some embodiments, the first initial enlarged specimen image is any one of the above-mentioned multiple initial enlarged specimen images, and the second processing unit 902 uses a super-resolution reconstruction algorithm to process each initial enlarged specimen image to obtain multiple target enlarged specimen images corresponding to the initial specimen image. Specifically, the method may include: the second processing unit 902 is used to extract the first image feature of the first initial enlarged specimen image through a pre-trained image super-resolution reconstruction network, and perform residual processing on the first image feature to obtain a residual processing result; up-sample the first image feature to obtain a first up-sampling result, and up-sample the residual processing result to obtain a second up-sampling processing result; obtain an up-sampling fusion result based on the first up-sampling result and the second up-sampling processing result, and perform super-resolution reconstruction on the up-sampling fusion result to obtain a target enlarged specimen image corresponding to the initial specimen image.

[0186] The second processing unit 902 is used to process the initial specimen image using an instance segmentation algorithm to obtain the recognition results and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image. Specifically, the second processing unit 902 is used to extract the structural feature image and the pixel feature image of the initial specimen image using a pre-trained image instance segmentation model; and, in the image instance segmentation model, determine the recognition result of each internal structure of the specimen indicated by the initial specimen image and the first position of each internal structure on the initial specimen image according to the structural feature image; and, in the image instance segmentation model, generate a background mask image, a foreground mask image and a transition zone mask image between the background and the foreground of each internal structure corresponding to each internal structure according to the pixel feature image, and generate a structural segmentation mask image corresponding to each internal structure according to the pixel feature image, the background mask image, the foreground mask image and the transition zone mask image; and use the structural segmentation mask image corresponding to each internal structure to correct the first position of each internal structure on the initial specimen image to obtain the pixel coordinate information of each internal structure.

[0187] In some embodiments, the second processing unit 902 is further used to obtain pixel coordinate information and recognition results of each internal structure of the specimen indicated by the initial specimen image according to the category and position of each internal structure and the structural segmentation mask image, and then determine the category of the specimen indicated by the initial specimen image from a specimen category library according to the recognition results of each internal structure of the specimen indicated by the initial specimen image; wherein the specimen category library includes multiple specimen categories and the recognition results of each internal structure of each specimen category.

[0188] See also Fig.10 , Fig.10 This is a structural block diagram of a terminal device disclosed in an embodiment of the present application. It includes:

[0189] A memory 1001 storing executable program codes;

[0190] A processor 1002 coupled to the memory 1001;

[0191] The processor 1002 calls the executable program code stored in the memory 1001 to execute the method in the above embodiment.

[0192] See also Fig.11 , Fig.11 This is a structural block diagram of a server disclosed in an embodiment of the present application. It includes:

[0193] A memory 1101 storing executable program codes;

[0194] A processor 1102 coupled to the memory 1101;

[0195] The processor 1102 calls the executable program code stored in the memory 1101 to execute the method in the above embodiment.

[0196] The embodiments of the present application disclose a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method in the above embodiments.

[0197] The embodiments of the present application disclose a computer program product. When the computer program product is run on a computer, the computer executes the method in the above embodiments.

[0198] The embodiment of the present application discloses an application publishing system, which is used to publish a computer program product. When the computer program product runs on a computer, the computer executes the method in the above embodiment.

[0199] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0200] The above is a detailed introduction to a specimen image processing method and terminal equipment, server, and storage medium disclosed in the embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The size of the step numbers in the above specific examples does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above units described as separate components may or may not be physically separated, and some or all of them may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0201] In this article, the character " / " generally indicates that the objects associated before and after are in an "or" relationship. In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information. In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on such an understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the technical solution or all or part of the technical solution, which can be embodied in the form of a software product, which is stored in a memory, including several requests to enable a computer device (which can be a personal computer, a server or a network device, etc., and can be a processor in a computer device) to perform some or all of the steps of the above methods of each embodiment of the present application.

[0202] The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A specimen image processing method, characterized in that: The method is applicable to a terminal device, the terminal device is equipped with a convex lens accessory, and the terminal device is communicatively connected with a server, and the method comprises: Using the camera of the terminal device to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image, and sending the initial specimen image to the server so that the server processes the initial specimen image to obtain a specimen database corresponding to the initial specimen image; The method of using the camera of the terminal device to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image includes: Using the camera of the terminal device, continuously photographing the glass slide placed on the convex lens accessory within a preset time period to obtain multiple frames of specimen images; Calculate the image difference value between each two adjacent frames in the multiple frames of specimen images by using a hash algorithm; If the image difference values ​​of each two adjacent frames in the multiple frames of specimen images are all less than the image difference threshold, it is determined that the glass slide is in a stationary state; Using a specimen image of any frame among the multiple frames of specimen images as an initial specimen image; receiving the specimen database sent by the server; wherein the specimen database stores specimen data, the specimen data including a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image; When an operation instruction for a current specimen image displayed on the screen of the terminal device is detected, specimen data matching the operation instruction is acquired according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image; The sample data matching the operation instruction is displayed on the screen.

2. The method according to claim 1, characterized in that The operation instruction includes a first amplification instruction, and acquiring specimen data matching the operation instruction according to the specimen database includes: Obtaining a first magnification indicated by the first magnification instruction; A first enlarged specimen image corresponding to the first magnification is obtained according to the plurality of target enlarged specimen images.

3. The method according to claim 1, characterized in that The operation instruction includes a structure selection instruction, and the specimen data also includes pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image; and acquiring the specimen data matching the operation instruction according to the specimen database includes: Determining a first selected area in the current specimen image according to the structure selection instruction; Determine, according to the first selected area, first pixel coordinate information corresponding to the first selected area in the initial specimen image; Determining a first internal structure corresponding to the first pixel coordinate information according to pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image; The recognition result of the first internal structure is acquired from the recognition results of each internal structure of the specimen indicated by the initial specimen image.

4. The method according to claim 3, characterized in that The step of determining first pixel coordinate information corresponding to the first selected area in the initial specimen image according to the first selected area includes: When the current specimen image is the initial specimen image, taking the pixel coordinate information corresponding to the first selected area as the first pixel coordinate information; When the current specimen image is an enlarged image of the initial specimen image, a first pixel coordinate mapping relationship between the current specimen image and the initial specimen image is determined according to a magnification ratio of the current specimen image relative to the initial specimen image; and according to the first pixel coordinate mapping relationship, second pixel coordinate information of the first selected area in the current specimen image is converted into first pixel coordinate information in the initial specimen image.

5. The method according to claim 3 or 4, characterized in that: The operation instruction further includes a second zoom-in instruction, and the structure selection instruction, after determining the first selected area in the current specimen image, the method further includes: Acquire a second magnification indicated by the second magnification instruction; Obtaining a second enlarged specimen image corresponding to the second magnification according to the plurality of target enlarged specimen images; According to the pixel coordinate information corresponding to the first selected area in the current specimen image and the second pixel coordinate mapping relationship between the current specimen image and the second enlarged specimen image, an enlarged image of the first internal structure is obtained from the second enlarged specimen image.

6. The method according to claim 5, characterized in that The method of obtaining a second enlarged specimen image corresponding to the second magnification by enlarging the specimen images according to the multiple targets includes: When there is a second enlarged specimen image corresponding to the second magnification in the plurality of target enlarged specimen images, acquiring the second enlarged specimen image from the plurality of target enlarged specimen images; In the case that there is no second enlarged specimen image corresponding to the second magnification among the multiple target enlarged specimen images, an enlarged specimen image with a magnification smaller than the second magnification and closest to the second magnification is used as a target specimen image among the multiple target enlarged specimen images; and the target specimen image is enlarged by using an interpolation algorithm to obtain a second enlarged specimen image corresponding to the second magnification.

7. The method according to any one of claims 1 to 4 and 6, characterized in that: The multiple target magnified specimen images corresponding to the initial specimen image are obtained by the server processing the initial specimen image using a super-resolution reconstruction algorithm; the recognition results of the internal structures of the specimen indicated by the initial specimen image are obtained by the server processing the initial specimen image using an instance segmentation algorithm; The specimen data also includes specimen category information of the specimen indicated by the initial specimen image.

8. A specimen image processing method, characterized in that: The method is applicable to a server, the server is connected to a terminal device, and the terminal device is equipped with a convex lens accessory, comprising: Receive an initial specimen image sent by the terminal device, wherein the initial specimen image is obtained by the terminal device using a camera of the terminal device to continuously photograph a glass slide placed on the convex lens accessory within a preset time period to obtain multiple frames of specimen images; use a hash algorithm to calculate an image difference value of each two adjacent frames in the multiple frames of specimen images; if the image difference value of each two adjacent frames in the multiple frames of specimen images is less than an image difference threshold, it is determined that the glass slide is in a stationary state; and obtain a specimen image of any frame in the multiple frames of specimen images as the initial specimen image; Processing the initial specimen image to obtain specimen data corresponding to the initial specimen image, and constructing a specimen database corresponding to the initial specimen image based on the specimen data, wherein the specimen data includes a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image; The specimen database is sent to the terminal device so that when the terminal device detects an operation instruction for a current specimen image displayed on a screen of the terminal device, the terminal device acquires specimen data matching the operation instruction according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image.

9. The method according to claim 8, characterized in that The specimen data also includes pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image; the processing of the initial specimen image to obtain the specimen data corresponding to the initial specimen image includes: The initial specimen image is magnified according to a preset magnification to obtain a plurality of initial magnified specimen images corresponding to the initial specimen image, and each initial magnified specimen image is processed by a super-resolution reconstruction algorithm to obtain a plurality of target magnified specimen images corresponding to the initial specimen image; The initial specimen image is processed by using an instance segmentation algorithm to obtain recognition results and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image.

10. The method according to claim 9, characterized in that The first initial enlarged specimen image is any one of the multiple initial enlarged specimen images, and the super-resolution reconstruction algorithm is used to process each initial enlarged specimen image to obtain multiple target enlarged specimen images corresponding to the initial specimen image, including: Extracting a first image feature of the first initial enlarged specimen image through a pre-trained image super-resolution reconstruction network, and performing residual processing on the first image feature to obtain a residual processing result; Performing upsampling processing on the first image feature to obtain a first upsampling result, and performing upsampling processing on the residual processing result to obtain a second upsampling processing result; An upsampling fusion result is obtained according to the first upsampling result and the second upsampling processing result, and super-resolution reconstruction is performed on the upsampling fusion result to obtain a target enlarged specimen image corresponding to the initial specimen image.

11. The method according to claim 9 or 10, characterized in that: The initial specimen image is processed by an instance segmentation algorithm to obtain the recognition result and pixel coordinate information of each internal structure of the specimen indicated by the initial specimen image, including: Extracting a structural feature image and a pixel feature image of the initial specimen image using a pre-trained image instance segmentation model; In the image instance segmentation model, determining, according to the structural feature image, a recognition result of each internal structure of the specimen indicated by the initial specimen image and a first position of each internal structure on the initial specimen image; In the image instance segmentation model, a background mask image, a foreground mask image and a transition zone mask image between the background and the foreground of each internal structure corresponding to each internal structure are generated according to the pixel feature image, and a structure segmentation mask image corresponding to each internal structure is generated according to the pixel feature image, the background mask image, the foreground mask image and the transition zone mask image; The structure segmentation mask image corresponding to each internal structure is used to correct the first position of each internal structure on the initial specimen image to obtain pixel coordinate information of each internal structure.

12. The method according to claim 11, characterized in that After obtaining pixel coordinate information and recognition results of each internal structure of the specimen indicated by the initial specimen image according to the category and position of each internal structure and the structure segmentation mask image, the method further includes: According to the recognition results of each internal structure of the specimen indicated by the initial specimen image, the specimen category corresponding to the initial specimen image is determined from a specimen category library; wherein the specimen category library includes multiple specimen categories and the recognition results of each internal structure of each specimen category.

13. A terminal device, characterized in that: The terminal device is equipped with a convex lens accessory and is connected to the server for communication, including: A collection unit, used to use the camera of the terminal device to photograph the glass slide placed on the convex lens accessory to obtain an initial specimen image; The acquisition unit is used to use the camera of the terminal device to photograph the glass slide placed on the convex lens accessory to obtain the initial specimen image, which specifically includes: The acquisition unit is used to use the camera of the terminal device to continuously shoot the glass slide placed on the convex lens accessory within a preset time length to obtain multiple frames of specimen images; and, using a hash algorithm, calculate the image difference value of each two adjacent frames in the multiple frames of specimen images; and, if the image difference value of each two adjacent frames in the multiple frames of specimen images is less than an image difference threshold, it is determined that the glass slide is in a stationary state; and, use the specimen image of any frame in the multiple frames of specimen images as an initial specimen image; A first transceiver unit is used to send the initial specimen image to the server so that the server processes the initial specimen image to obtain a specimen database corresponding to the initial specimen image; and receive the specimen database sent by the server; wherein the specimen database stores specimen data, the specimen data including a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image; A first processing unit is configured to, when an operation instruction for a current specimen image displayed on a screen of the terminal device is detected, acquire specimen data matching the operation instruction according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image; The display unit is used to display the specimen data matching the operation instruction on the screen.

14. A terminal device, characterized in that: include: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the method according to any one of claims 1 to 7.

15. A server, characterized in that: The server is connected to a terminal device, and the terminal device is equipped with a convex lens accessory, including: A second transceiver unit is used to receive an initial specimen image sent by the terminal device, wherein the initial specimen image is obtained by the terminal device using a camera of the terminal device to continuously photograph a glass slide placed on the convex lens accessory within a preset time period to obtain multiple frames of specimen images; using a hash algorithm to calculate an image difference value of each two adjacent frames in the multiple frames of specimen images; if the image difference value of each two adjacent frames in the multiple frames of specimen images is less than an image difference threshold, it is determined that the glass slide is in a stationary state; and a specimen image of any frame in the multiple frames of specimen images is obtained as the initial specimen image; a second processing unit, configured to process the initial specimen image to obtain specimen data corresponding to the initial specimen image, and to construct a specimen database corresponding to the initial specimen image according to the specimen data, wherein the specimen data includes a plurality of target magnified specimen images corresponding to the initial specimen image, and recognition results of various internal structures of the specimen indicated by the initial specimen image; The second transceiver unit is also used to send the specimen database to the terminal device, so that when the terminal device detects an operation instruction for the current specimen image displayed on the screen of the terminal device, the terminal device obtains specimen data matching the operation instruction according to the specimen database; wherein the current specimen image is the initial specimen image or an enlarged image of the initial specimen image.

16. A server, characterized in that: include: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the method according to any one of claims 8 to 12.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 12 is implemented.

Images