Image recognition method, device, terminal device and storage medium

Abstract

The present application discloses an image recognition method, apparatus, terminal device and storage medium, belonging to the field of image processing technology. Applied to a terminal device, the terminal device includes a camera and a detachable slide assembly, the camera is facing the detachable slide assembly, the slide assembly includes a slide holder, the slide holder is used to place the slide, the method includes: obtaining a target specimen image captured by the camera; performing image recognition on the target specimen image to obtain an image recognition result; determining that the image currently displayed on the display screen corresponds to the image area position of the target specimen image; obtaining the target recognition result corresponding to the image area position according to the image recognition result, and displaying the target recognition result on the display screen. The terminal device of the present application can recognize the captured target specimen image in real time, display the target recognition result on the display screen, and improve the efficiency of the terminal device in obtaining information on the target specimen image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image recognition method, apparatus, terminal device and storage medium. Background Art

[0002] With the development of science and technology, microscopes have been widely used as a device for observing objects. People can use microscopes to view the finer structures of objects.

[0003] Currently, in order to facilitate people's use, more and more terminal devices can be installed with cameras. Users can combine this camera with a slide holder to form a microscope, so that the terminal device can also obtain the function of obtaining the more subtle structure of the object. In this microscope structure, the user places a slide carrying a specimen on the slide holder, and then controls the camera to shoot through the terminal device. Among them, if the user needs to understand the image captured by the microscope, he needs to use his own experience to understand the microscopic structure of the object, obtain information such as the category of the currently observed image, or obtain information about similar objects by consulting relevant books.

[0004] In the above-mentioned related technologies, since the terminal device fails to display the information of the observed image in a timely manner, it takes a long time and is inefficient to obtain the object information corresponding to the specimen. Summary of the Invention

[0005] The embodiments of the present application provide an image recognition method, apparatus, terminal device, and storage medium, which can improve the efficiency of the terminal device in acquiring information of a target specimen image.

[0006] On the one hand, an embodiment of the present application provides an image recognition method, which is applied to a terminal device, wherein the terminal device includes a camera and a detachable slide assembly, the camera facing the detachable slide assembly, the slide assembly including a slide holder, and the slide holder is used to place a slide, the method comprising:

[0007] Acquiring a target specimen image captured by the camera, wherein the target specimen image is an image of the specimen contained on the slide at a basic magnification;

[0008] performing image recognition on the target specimen image to obtain an image recognition result;

[0009] Determining that the image currently displayed on the display screen corresponds to the image area position of the target specimen image;

[0010] The target recognition result corresponding to the position of the image area is obtained according to the image recognition result, and the target recognition result is displayed on the display screen.

[0011] Optionally, the image recognition result includes the specimen category of the specimen, the structural category of the component structure of the specimen, and the pixel coordinates of the component structure in the target specimen image. The image recognition of the target specimen image to obtain the image recognition result includes:

[0012] Inputting the target specimen image into a first recognition model, recognizing the target specimen image using the first recognition model to obtain a specimen category corresponding to the target specimen image, wherein the first recognition model is trained based on a first sample image set, wherein the first sample image set includes a plurality of first specimen images at a basic magnification, each first specimen image carrying a corresponding specimen category label;

[0013] The target specimen image is input into a second recognition model, and the target specimen image is recognized by the second recognition model to obtain a structural category of a component structure contained in the target specimen image and a pixel coordinate of the component structure in the target specimen image. The second recognition model is trained based on a second sample image set, and the second sample image set includes multiple second specimen images at a basic magnification, each second specimen image carrying a structural category label of a corresponding component structure and a corresponding pixel coordinate label.

[0014] Optionally, determining the position of the image frame currently displayed on the display screen in the image area of ​​the target specimen image includes:

[0015] Determining the magnification and movement information of the digital magnifying glass of the terminal device, wherein the movement information is the displacement of the target specimen image in the image coordinate system of the display screen after the target specimen image is magnified according to the magnification;

[0016] performing coordinate conversion on pixel coordinates of the target specimen image according to the magnification and the movement information to obtain image coordinates of the target specimen image in an image coordinate system on the display screen;

[0017] The position of the image region corresponding to the image currently displayed on the display screen in the target specimen image is determined according to the image coordinates in the image coordinate system.

[0018] Optionally, acquiring a target recognition result corresponding to the position of the image area according to the image recognition result, and displaying the target recognition result on the display screen includes:

[0019] Filtering the target recognition result from the image recognition result according to the image area position and the image recognition result, the target recognition result including target display coordinates, the specimen category corresponding to the target display coordinates, and the structural category of the component structure at the target display coordinates, the target display coordinates being the pixel coordinates contained in the image area position;

[0020] The target recognition result is displayed in the display area of ​​the display screen according to a preset display method.

[0021] Optionally, before acquiring the target specimen image captured by the camera, the method further includes:

[0022] Detecting whether the magnification of the digital magnifying glass of the terminal device is a basic magnification;

[0023] When the magnification of the digital magnifying glass is the basic magnification, controlling the camera to photograph the glass slide placed on the glass slide holder to obtain a first specimen image, and using the first specimen image as the target specimen image;

[0024] When the magnification of the digital magnifying glass is not the basic magnification, a second specimen image of a pre-stored basic magnification is acquired, and the second specimen image is used as the target specimen image.

[0025] Optionally, before acquiring the target specimen image, the method further includes:

[0026] Acquire a current specimen image and previous N frames of specimen images obtained by photographing the glass slide placed on the glass slide holder with the camera, where N is a positive integer;

[0027] Calculating hash values ​​of the current specimen image and the previous N frames of specimen images;

[0028] When the differences between the hash values ​​corresponding to the specimen images of adjacent frames are all less than a preset difference threshold, the step of acquiring the target specimen image captured by the camera is performed.

[0029] Optionally, a target application is running in the terminal device, and an application interface corresponding to the target application includes an image of the specimen displayed on the display screen. Before acquiring the target specimen image, the method further includes:

[0030] Acquire the operating state of the target application, where the operating state includes an operating state and a non-operating state;

[0031] When the operation state of the target application is the non-operation state, the step of acquiring the target specimen image captured by the camera is performed.

[0032] On the other hand, an embodiment of the present application provides an image recognition device, which is applied to a terminal device, wherein the terminal device includes a camera and a detachable slide assembly, wherein the camera faces the detachable slide assembly, and the slide assembly includes a slide holder and a slide, and the device includes:

[0033] an image acquisition module, configured to acquire a target specimen image, wherein the target specimen image is an image of the specimen contained on the slide at a base magnification;

[0034] An image recognition module, configured to perform image recognition on the target specimen image to obtain an image recognition result;

[0035] The result acquisition module is used to obtain the target recognition result corresponding to the target position in the target specimen image from the image recognition result, where the target position is the image area position in the target specimen image corresponding to the image displayed on the display screen of the terminal device.

[0036] On the other hand, an embodiment of the present application provides a terminal device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor implements the image recognition method as described in one aspect above and any optional method thereof.

[0037] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image recognition method as described in the above aspect and any optional manner thereof.

[0038] On the other hand, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the image recognition method as described in the above aspect and any optional manner thereof.

[0039] On the other hand, an embodiment of the present application provides an application publishing platform, which is used to publish a computer program product, wherein when the computer program product is run on a computer, the computer executes the image recognition method as described in one of the above aspects and any optional method thereof.

[0040] The technical solutions provided by the embodiments of the present application may have at least the following beneficial effects:

[0041] The terminal device of the present application obtains the target specimen image captured by the camera, and the target specimen image is the image of the specimen contained on the slide at the basic magnification; performs image recognition on the target specimen image to obtain the image recognition result; determines that the image currently displayed on the display screen corresponds to the image area position of the target specimen image; obtains the target recognition result corresponding to the image area position based on the image recognition result, and displays the target recognition result on the display screen. The terminal device can perform image recognition based on the image at the basic magnification, and determine that the image currently displayed on the display screen corresponds to the image area position of the target specimen image, obtain the target recognition result corresponding to the image area position from the image recognition result and display it. It can recognize the captured target specimen image in real time and display the recognition result of the image currently displayed on the display screen, so that the terminal device can more conveniently display the specimen information and improve the efficiency of the terminal device in obtaining the information of the target specimen image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of a scene architecture for shooting by a terminal device provided by an exemplary embodiment of the present application;

[0043] Figure 2 is a method flow chart of an image recognition method provided by an exemplary embodiment of the present application;

[0044] Figure 3 is a schematic structural diagram of a target specimen image involved in an exemplary embodiment of the present application;

[0045] Figure 4 is a method flow chart of an image recognition method provided by an exemplary embodiment of the present application;

[0046] Figure 5 This is a schematic diagram of an image display interface according to an exemplary embodiment of the present application;

[0047] Figure 6 is a structural diagram of an image coordinate system involved in an exemplary embodiment of the present application;

[0048] Figure 7 This is a schematic diagram showing an image recognition result according to an exemplary embodiment of the present application;

[0049] Figure 8 This is a schematic diagram of a selection interface involved in an exemplary embodiment of the present application;

[0050] Figure 9 This is a schematic diagram of an interface for displaying target recognition results according to an exemplary embodiment of the present application;

[0051] Figure 10This is a schematic diagram of an interface for displaying target recognition results according to an exemplary embodiment of the present application;

[0052] Figure 11 is a method flow chart of an image recognition method provided by an exemplary embodiment of the present application;

[0053] Figure 12 is a structural block diagram of an image recognition device provided by an exemplary embodiment of the present application;

[0054] Figure 13 It is a structural diagram of a terminal device disclosed in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0056] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0057] The solution provided in this application can be used in real-life scenarios where people use terminal devices in their daily lives to capture images of objects on a slide fixed on a slide holder through a camera connected to the terminal device. For ease of understanding, the scenario architecture involved in the embodiments of this application is first briefly introduced below.

[0058] With the development of science and technology, various terminal devices have appeared in people's daily lives, allowing people to work, entertain, study, and so on. In terms of capturing more microscopic images of objects, improvements can now be made to terminal devices. By utilizing the terminal device's built-in camera module, images of even finer structures can be captured. For example, the terminal device can be equipped with a detachable slide assembly, with the terminal device's camera facing the detachable slide assembly. The slide assembly includes a slide holder on which a slide can be placed. When the slide assembly is installed on the terminal device, the slide can also face the camera module, allowing the terminal device to capture images of even finer structures of objects through the camera module.

[0059] Please refer to Figure 1 , which shows a schematic diagram of a scene architecture of a terminal device shooting provided by an exemplary embodiment of the present application. Figure 1 As shown, the scenario architecture includes a terminal device 110.

[0060] The terminal device 110 is a terminal that may be equipped with a camera. For example, the terminal may be a mobile phone, a tablet computer, an e-book reader, smart glasses, a smart watch, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a notebook computer, a portable computer, a desktop computer, etc.

[0061] By adding a detachable slide assembly 120 to the terminal device 110, the terminal device can realize the function of capturing images of finer structures of objects through a camera. The slide assembly 120 may include a slide holder 121, on which a slide 122 may be placed. After the user places the specimen on the slide 122, it can be installed on the slide holder 121, and then the slide assembly 120 as a whole is installed in a position facing the camera, so that the terminal device can observe the specimen on the slide through the camera. Optionally, the terminal device 110 can control the camera to shoot and obtain an image of the specimen displayed on the slide. That is, the camera and slide assembly on the terminal device can constitute a simple "microscope" to capture finer structures of the specimen.

[0062] In one possible implementation, the terminal device 110 may be connected to the slide assembly via a communication network, which may be a wired network or a wireless network. That is, the terminal device controls the slide assembly via the communication network, such as controlling the insertion and movement of the slide.

[0063] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network. In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0064] In the above scenario architecture, after placing the specimen on a slide and mounting the slide on a slide holder, the user can control the camera to shoot by controlling the terminal device, thereby acquiring an image of the specimen on the slide. Usually, the user observes the image of the captured specimen and obtains information about the observed specimen image through his or her own cognition of each object, such as the category to which the specimen belongs, the composition structure of the specimen, etc. At present, the terminal device only displays the specimen image captured by the camera on the display screen. There is no more complete technical solution for displaying the information of the specimen, which results in the terminal device not displaying the information of the observed image in a timely manner, resulting in a long time and low efficiency in obtaining the object information corresponding to the specimen.

[0065] In order to avoid the problems existing in the above technical solutions and improve the efficiency of the terminal device in timely obtaining the identification results of the specimen and displaying them on the display screen, this application provides an image recognition method that can enable the terminal device to actively control the camera to shoot and obtain the target identification results of the specimen, thereby improving the efficiency of displaying the specimen information.

[0066] Please refer to Figure 2 , which shows a method flow chart of an image recognition method provided by an exemplary embodiment of the present application. This method can be used for Figure 1In the scenario architecture shown, the terminal device in the scenario architecture is executed, and the terminal device includes a camera and a detachable slide assembly. The camera is facing the detachable slide assembly. The slide assembly includes a slide holder. The slide holder is used to place the slide, such as Figure 2 As shown, the image recognition method may include the following steps.

[0067] Step 201 : Acquire a target specimen image captured by a camera. The target specimen image is an image of the specimen contained on a glass slide at a basic magnification.

[0068] Optionally, the basic magnification is the magnification of the optical magnifying glass included in the slide assembly. The optical magnifying glass can be installed between the camera and the slide, and the image captured by the camera is the image after being magnified by the optical magnifying glass. The basic magnification can be pre-set by the developer (for example, 1x, 2x, 3x, etc.). The user can place the specimen on the slide, place the slide on the slide holder, position the optical magnifying glass correctly, and control the camera to shoot through the terminal device. After the camera captures the specimen image of the specimen on the slide based on the basic magnification, the terminal device can obtain the image, which is the target specimen image.

[0069] Alternatively, the specimen can be any object in real life that can be placed on a slide, for example, a petal, a leaf, a piece of muscle tissue, etc.

[0070] Step 202: Perform image recognition on the target specimen image to obtain an image recognition result.

[0071] Optionally, the terminal device can perform image recognition on the acquired target specimen image to obtain an image recognition result. This image recognition result may include the specimen category, the composition structure of the specimen, etc. For example, if the specimen is a petal, after performing image recognition on the target specimen image by the terminal device, it can be determined that the specimen category is a petal. The composition structure of the petal may include vacuoles, mitochondria, cell walls, etc., or may include the base, middle, or outer part of the petal.

[0072] Please refer to Figure 3 , which shows a schematic structural diagram of a target specimen image involved in an exemplary embodiment of the present application. Figure 3As shown, the petals 300 include a petal base 301, a petal middle 302, and a petal outer portion 303. Therefore, the terminal device can perform image recognition, and the image recognition result obtained includes the composition structure of the petal base 301, the petal middle 302, and the petal outer portion 303. It should be noted that the specific composition structure here can be pre-trained by the developer through a machine learning model, and this application does not limit this.

[0073] Step 203: Determine the image area position of the target specimen image corresponding to the image frame currently displayed on the display screen.

[0074] Optionally, the terminal device has a display screen, and the image currently displayed on the display screen displays the entire or partial image of the target specimen image. In this step, the terminal device can determine the image area position of the target specimen image corresponding to the image currently displayed on the display screen. The image area position is the coordinate area position in the pixel coordinate system of the target specimen image. In other words, the terminal device can determine the coordinate position area of ​​the target specimen image corresponding to the image currently displayed on the display screen in the pixel coordinate system, and display the coordinate position area of ​​the target specimen image in the pixel coordinate system in accordance with the display mode of the display screen.

[0075] Step 204 : Obtain the target recognition result corresponding to the position of the image area according to the image recognition result, and display the target recognition result on the display screen.

[0076] Optionally, the terminal device obtains the target recognition result corresponding to the image region position based on the image recognition result obtained above, and displays the target recognition result on the display screen to timely display relevant information about the specimen. The image recognition result obtained by the terminal device from performing image recognition on the target specimen image includes regional information, which can indicate the location of the respective structures in different regions of the target specimen image. If a partial image of the target specimen image is displayed on the display screen of the terminal device, the terminal device can simply display the image recognition result of the corresponding partial image.

[0077] In summary, the terminal device of the present application obtains the target specimen image captured by the camera, and the target specimen image is the image of the specimen contained on the slide at the basic magnification; performs image recognition on the target specimen image to obtain the image recognition result; determines that the image currently displayed on the display screen corresponds to the image area position of the target specimen image; obtains the target recognition result corresponding to the image area position according to the image recognition result, and displays the target recognition result on the display screen. The terminal device can perform image recognition based on the image at the basic magnification, and determine that the image currently displayed on the display screen corresponds to the image area position of the target specimen image, obtain the target recognition result corresponding to the image area position from the image recognition result and display it, and can recognize the captured target specimen image in real time, and display the recognition result of the image currently displayed on the display screen, so that the terminal device can more conveniently display the specimen information, and improve the efficiency of the terminal device in obtaining the information of the target specimen image.

[0078] In one possible implementation, a target application is installed in a terminal device, which may be an application used for observing specimens. The application interface corresponding to the target application includes an image of the specimen displayed on a display screen. Before the terminal device acquires the target specimen image, the terminal device may detect the operating status of the target application, which includes an operating state and a non-operating state. When the operating state of the target application is a non-operating state, the terminal device may acquire the target specimen image, thereby executing subsequent steps to make the displayed specimen information more stable.

[0079] Please refer to Figure 4 , which shows a method flow chart of an image recognition method provided by an exemplary embodiment of the present application. This method can be used for Figure 1 In the scenario architecture shown, the terminal device in the scenario architecture is executed, and the terminal device includes a camera and a detachable slide assembly. The camera is facing the detachable slide assembly. The slide assembly includes a slide holder. The slide holder is used to place the slide, such as Figure 4 As shown, the image recognition method may include the following steps.

[0080] Step 401: Acquire the operating status of the target application, where the operating status includes an operating status and a non-operating status.

[0081] Optionally, the terminal device may have a target application pre-installed, which may be an application that needs to be launched when observing a specimen. For example, when a user needs to observe a specimen through the terminal device, they need to launch the target application, display a corresponding application interface in the target application, and display the specimen image captured by the camera in the application interface.

[0082] Optionally, the operating state of the target application includes an operating state and a non-operating state. The operating state means that the user causes the target application to run other processes, etc. through touch operations, mouse clicks, etc. For example, the user controls the movement of the image screen on the display screen through touch operations. At this time, the operating state of the target application is the operating state. Among them, the developer of the target application can monitor whether the target application is in the operating state by embedding points at one or more nodes in the application, thereby obtaining the operating state of the target application. Alternatively, the terminal device can also determine the operating state of the target application by detecting the trigger operation in the display screen. This application does not limit the method of how to obtain the operating state of the target application.

[0083] Step 402: When the target application is in an inoperative state, an image of the target specimen captured by a camera is acquired.

[0084] The target specimen image is an image of the specimen contained on the slide at a basic magnification. Optionally, the basic magnification is the magnification of the optical magnifier contained in the slide assembly. In the embodiment of the present application, the basic magnification may be 1x magnification.

[0085] When the target application's operating state is not in operation, the terminal device may execute the step of acquiring a target specimen image captured by a camera. The terminal device may acquire the target specimen image by calling the camera to capture a glass slide placed on a glass slide holder.

[0086] In one possible implementation, in order to improve the accuracy of acquiring the target specimen image, during the shooting process, the terminal device can also obtain the current specimen image and the previous N frames of specimen images obtained by the camera shooting the slide placed on the slide holder, where N is a positive integer; calculate the hash values ​​of the current specimen image and the previous N frames of specimen images; when the difference between the hash values ​​corresponding to the specimen images of adjacent frames is less than a preset difference threshold, execute the step of acquiring the target specimen image captured by the camera.

[0087] That is, before each acquisition of a target specimen image, the terminal device can acquire the current frame image and the previous N frames of specimen images. For example, the terminal device acquires multiple frames of specimen images within A seconds and calculates the hash value of each frame. If the difference between the hash values ​​corresponding to the specimen images of adjacent frames is less than a preset difference threshold, the current frame image is acquired as the target specimen image. The preset difference threshold can be pre-set by the developer. For example, the preset difference threshold is B. When the terminal device prepares to acquire the target specimen image, it calculates the hash values ​​of the current specimen image and the previous N frames of specimen images, and obtains the difference between the hash values ​​corresponding to the specimen images of adjacent frames. If the difference between the hash values ​​of each adjacent frame of specimen images is less than B, it indicates that the slide has not moved or shaken at this time, and the image acquired from the specimen on the slide is stable. At this time, the terminal device can use the acquired current specimen image as the target specimen image.

[0088] In one possible implementation, before acquiring the target specimen image, the terminal device may further detect whether the magnification of the digital magnifying glass of the terminal device is a basic magnification; when the magnification of the digital magnifying glass is the basic magnification, the camera is controlled to photograph the slide placed on the slide holder to obtain a first specimen image, and the first specimen image is used as the target specimen image; when the magnification of the digital magnifying glass is not the basic magnification, a second specimen image of the basic magnification that is pre-stored is obtained, and the second specimen image is used as the target specimen image.

[0089] The magnification of the digital magnifying glass can be regarded as the magnification of any image by the terminal device when the terminal device uses the target application to magnify the image. For example, the terminal device can display the image on the display screen and perform operations such as zooming in or out on the image. The digital magnifying glass is a virtual magnifying glass in the terminal device, and the zoom-in or zoom-out ratio is the magnification of the digital magnifying glass. Please refer to Figure 5 , which shows a schematic diagram of an image display interface involved in an exemplary embodiment of the present application. Figure 5 As shown, the terminal device 500 includes a display area 501 and a mask area 502. The terminal device 500 displays an image in the display area 501. The user can zoom in or out the image in the display area 501 by operating in the display area or operating the virtual controls in the interface, thereby achieving a digital zoom effect.

[0090] As mentioned above Figure 5As shown, the terminal device can display the specimen image on the display screen through the target application. The terminal device detects whether the magnification of the digital magnifying glass is the basic magnification. Here, the basic magnification is also taken as the above-mentioned 1x magnification. The terminal device detects whether the magnification of the digital magnifying glass is 1x. If the magnification of the digital magnifying glass is 1x, the terminal device can control the camera to capture the slide placed on the slide holder to obtain a first specimen image, and use the first specimen image as the target specimen image, thereby acquiring the target specimen image. If the magnification of the digital magnifying glass is not 1x, the terminal device can obtain a second specimen image of the basic magnification pre-stored from the preset memory, and use the second specimen image as the target specimen image. The second specimen image can be a sample that the developer has previously collected and stored in the preset memory for the same specimen.

[0091] In one possible implementation, before starting the target application, the user can also place a specimen on a slide and mount the slide on a slide holder. Optionally, a trigger device can be installed on the slide holder, and the terminal device detects whether the slide is placed through the trigger device. When the slide is mounted on the slide holder, the trigger device can be triggered to feedback a trigger signal to the terminal device. The trigger signal can indicate that the slide has been mounted on the slide holder, so that the terminal device detects the placement of the slide according to the trigger signal. For example, a Hall device is installed in the receiving slot on the slide holder for placing the slide. When the user places the slide in the receiving slot on the slide holder, the slide can trigger the Hall device, so that the terminal device receives the trigger signal fed back by the Hall device, thereby instructing the terminal device that the slide is placed on the slide holder, and triggering the terminal device to detect the operating status of the target application.

[0092] Optionally, when the user moves the slide and observes the specimen at a different angle, the terminal device can re-execute steps 401 to 402 to acquire a new target specimen image. For example, after the user moves the slide, the terminal device can receive a trigger signal fed back by the Hall effect device, causing the terminal device to re-execute the above steps for the moved specimen.

[0093] Step 403: Perform image recognition on the target specimen image to obtain an image recognition result.

[0094] The image recognition result includes the specimen category of the specimen, the structural category of the component structure of the specimen, and the pixel coordinates of the component structure in the target specimen image.

[0095] Optionally, the specimen category of the specimen can be the class to which the specimen belongs, for example, leaves, trunks, peels, petals, grass, hair, etc. The accuracy of the category to which it belongs is determined by the trained model and will not be repeated here. Optionally, the terminal device can input the target specimen image into the first recognition model, recognize the target specimen image through the first recognition model, and obtain the specimen category corresponding to the target specimen image. Among them, the first recognition model is obtained by training based on the first sample image set, and the first sample image set contains multiple first specimen images at a basic magnification, and each first specimen image carries a corresponding specimen category label. That is, the developer can pre-train the model with each sample image of the first sample image set to obtain a machine learning model that can recognize the specimen category of each sample image, thereby obtaining the first recognition model, and setting the first recognition model in the terminal device to achieve the effect of image recognition of the target specimen image.

[0096] For example, the specimen category labels corresponding to each first specimen image contained in the first sample image set are leaves, trunks, peels, petals, grass and other labels. Developers train the model through the first sample image set to obtain a first recognition model. By inputting an image into the first recognition model, the first recognition model can output the result of the specimen category corresponding to the image.

[0097] Optionally, the terminal device may also input the target specimen image into a second recognition model, and use the second recognition model to recognize the target specimen image, thereby obtaining the structural category of the component structure contained in the target specimen image, as well as the pixel coordinates of the component structure in the target specimen image. The second recognition model is trained based on a second sample image set, which includes multiple second specimen images at a base magnification, each of which carries a corresponding structural category label for the component structure and a corresponding pixel coordinate label. Image recognition is performed on the target specimen image to obtain an image recognition result.

[0098] Similarly, the structural category labels of the constituent structures corresponding to each second specimen image contained in the second sample image set are chloroplasts, mitochondria, cell walls, vacuoles, etc., and the corresponding pixel coordinate labels are area A, area B, area C, etc. The developers train the model through the second sample image set to obtain a second recognition model. By inputting an image into the second recognition model, the first recognition model can output the result of the structural category of the constituent structure corresponding to the image, as well as the result of the area within the image where the constituent structure is located.

[0099] Optionally, the effects achieved by the above-mentioned first recognition model and second recognition model can also be achieved through one or more machine learning models, which is not limited in the embodiments of the present application.

[0100] Step 404 : determining the magnification and movement information of the digital magnifying glass of the terminal device. The movement information is the displacement of the target specimen image in the image coordinate system of the display screen after the target specimen image is magnified according to the magnification.

[0101] Optionally, the terminal device can obtain the magnification of the digital magnifier currently displaying the target specimen image through the target application. For example, if the user magnifies the target specimen image by 10 times using the magnification control while viewing the target specimen image, the magnification of the digital magnifier obtained by the terminal device is 10 times.

[0102] Optionally, when viewing the target specimen image, the user can move the target specimen image by dragging the control, and the terminal device can record the displacement of the user's movement based on the image coordinate system of the display screen. Figure 6 , which shows a schematic structural diagram of an image coordinate system involved in an exemplary embodiment of the present application. Figure 6 As shown, the image coordinate system 600 is a coordinate system established by the terminal device on the display screen through the target application. The user controls the movement of the displayed target specimen image through the display screen 602 of the terminal device 601. For example, if the finger moves from the dotted line position to the solid line position, the user's movement distance of the target specimen image on the horizontal and vertical coordinate axes can be recorded by the terminal device, so that the terminal device can obtain movement information.

[0103] Step 405 : performing coordinate transformation on the pixel coordinates of the target specimen image according to the magnification and the movement information, and obtaining the image coordinates of the target specimen image in the image coordinate system on the display screen.

[0104] Optionally, after obtaining the magnification and movement information, the terminal device amplifies, moves, and processes the target specimen image to obtain the result of the magnified target specimen image on the simulated display screen, thereby realizing coordinate conversion and obtaining the image coordinates of the target specimen image in the image coordinate system on the display screen.

[0105] Step 406 : Determine the position of the image region in the target specimen image corresponding to the image currently displayed on the display screen according to the image coordinates in the image coordinate system.

[0106] Optionally, when the terminal device is displaying the target specimen image, the user displays the entire or partial image of the target specimen in the display area through operations such as zooming in and moving. After the terminal device obtains the image coordinates of the target specimen image in the image coordinate system on the display screen based on the magnification and movement information, it can select a position area corresponding to the image picture currently displayed on the display screen from the image coordinates in the image coordinate system, and determine the position of the image area in the target specimen image through the position area.

[0107] For example, the terminal device obtains the corresponding relationship of the coordinate transformation of the pixel coordinates in the target specimen image before obtaining the coordinate transformation, thereby obtaining the image area position in the target specimen image corresponding to the image frame currently displayed on the display screen in the image coordinate system. Alternatively, the terminal device can also simulate the display area in the display screen to a corresponding preset acquisition area through the target application, intercept the target specimen image in the image coordinate system through the preset acquisition area, obtain the corresponding position area, and convert the image coordinates in the image coordinate system to the image area position in the target specimen image before magnification and movement, thereby obtaining the image area position in the target specimen image corresponding to the image frame currently displayed on the display screen.

[0108] For example, the target specimen image is magnified according to the acquired magnification factor, and the displacement information corresponding to the center position of the magnified target specimen image is moved. The magnified and moved target specimen image is captured using a preset acquisition area, thereby determining the image coordinates within the corresponding area to be displayed on the display screen. The image coordinates are then converted back to the unmagnified, unmoved target specimen image to obtain the image area position corresponding to the image coordinates in the target specimen image. The preset acquisition area is the same size as the display area on the display screen, and the center of the preset acquisition area coincides with the center position of the target specimen image before the movement.

[0109] Step 407, based on the image area position and the image recognition result, the target recognition result is filtered out from the image recognition result, the target recognition result includes the target display coordinates, the specimen category corresponding to the target display coordinates, and the structural category of the component structure at the target display coordinates. The target display coordinates are the pixel coordinates contained in the image area position.

[0110] The terminal device filters the image recognition results according to the image area position to obtain the target recognition results that need to be displayed in the image recognition results.

[0111] Optionally, the above-mentioned image recognition results include the specimen category of the specimen identified by the first recognition model, the structural category of the component structure of the specimen identified by the second recognition model, and the pixel coordinates of the component structure in the target specimen image. In this step, the target recognition results are filtered out from the image recognition results by determining the pixel coordinates corresponding to the image area position.

[0112] For example, see Figure 7 , which shows a schematic diagram of an image recognition result according to an exemplary embodiment of the present application. Figure 7As shown, in the target specimen image 700, the specimen's component structure 1 701, the specimen's component structure 2 702, and the specimen's component structure 3 703 are included. The structural categories corresponding to the various component structures are category 1, category 2, and category 3, respectively. The pixel coordinates of the specimen's component structure 1 701 in the target specimen image can be represented by pixel coordinate 1, the pixel coordinates of the specimen's component structure 2 702 in the target specimen image can be represented by pixel coordinate 2, and the pixel coordinates of the specimen's component structure 3 703 in the target specimen image can be represented by pixel coordinate 3. The terminal device obtains the specimen category corresponding to the target specimen image 700 through the above-mentioned first recognition model, and obtains the structural category corresponding to each component structure and the pixel coordinates of the component structure in the target specimen image through the second recognition model. The image area position includes pixel coordinate 1 and pixel coordinate 2. The terminal device knows from the image area position that pixel coordinate 1 and pixel coordinate 2 are the target display coordinates, and obtains the specimen category corresponding to the target display coordinates and the structural category of the component structure at the target display coordinates from the image recognition result through the target display coordinates.

[0113] Step 408: Display the target recognition result in the display area of ​​the display screen according to a preset display method.

[0114] Optionally, the terminal device may display the obtained target recognition result in a display area of ​​the display screen according to a preset display method.

[0115] In one possible implementation, the preset display mode can be selected by the user in the target application selection interface. For example, please refer to Figure 8 , which shows a schematic diagram of a selection interface involved in an exemplary embodiment of the present application. Figure 8 As shown, in the display mode selection interface 800 of the terminal device, there are a first display mode 801, a second display mode 802, and a third display mode 803. The user can determine how the terminal device displays the target recognition result in the display area of ​​the display screen by selecting the corresponding display mode.

[0116] For example, the first display mode 801 is to display in a table format. Figure 8 After selecting the first display mode, the terminal device can display the target recognition results in the display area of ​​the display screen in the form of a table. The second display mode 802 is to display in the form of an arrow. Figure 8 After selecting the second display mode, the terminal device can display the target recognition results in the display area of ​​the display screen in the form of arrows. Figure 9 , which shows a schematic diagram of an interface for displaying target recognition results involved in an exemplary embodiment of the present application. Figure 9As shown, the display interface 900 includes the target recognition result 901 to be displayed. The third display mode 803 is to display in the form of tags. Figure 8 After selecting the third display mode, the terminal device can display the target recognition results in the display area of ​​the display screen in the form of labels. Figure 10 , which shows a schematic diagram of an interface for displaying target recognition results involved in an exemplary embodiment of the present application. Figure 10 As shown, the display interface 1000 includes the target recognition result 1001 to be displayed.

[0117] In summary, the terminal device of the present application obtains the target specimen image captured by the camera, and the target specimen image is the image of the specimen contained on the slide at the basic magnification; performs image recognition on the target specimen image to obtain the image recognition result; determines that the image currently displayed on the display screen corresponds to the image area position of the target specimen image; obtains the target recognition result corresponding to the image area position according to the image recognition result, and displays the target recognition result on the display screen. The terminal device can perform image recognition based on the image at the basic magnification, and determine that the image currently displayed on the display screen corresponds to the image area position of the target specimen image, obtain the target recognition result corresponding to the image area position from the image recognition result and display it, and can recognize the captured target specimen image in real time, and display the recognition result of the image currently displayed on the display screen, so that the terminal device can more conveniently display the specimen information, and improve the efficiency of the terminal device in obtaining the information of the target specimen image.

[0118] In addition, the second recognition model combines the pixel coordinates of the constituent structures when identifying the structural category of the constituent structures, making the subsequent display of the recognition results more accurate and improving the accuracy of the information displayed. This application also avoids image acquisition of the specimen in an unstable state by detecting the operating status of the target application and whether the specimen on the slide has moved, thereby improving the accuracy of specimen identification and specimen structure detection and the real-time performance of the overall system.

[0119] In the following, the terminal device is a tablet computer, and the tablet computer is combined with its own camera and a detachable slide assembly as an example. Figure 2 and Figure 4 The embodiment shown is used as an example. Figure 11 , which shows a method flow chart of an image recognition method provided by an exemplary embodiment of the present application. This method can be used for Figure 1In the scene architecture shown, the operation is performed by a tablet computer in the scene architecture, which includes a camera and a detachable slide assembly. The camera faces the detachable slide assembly. The slide assembly includes a slide holder, which is used to place the slide. Figure 11 As shown, the image recognition method may include the following steps.

[0120] Step 1101: Install a glass slide assembly at a position corresponding to the camera of the tablet computer.

[0121] Optionally, the user installs the glass slide assembly at a corresponding position of the camera according to a specified position, so that the camera can capture the image of the glass slide.

[0122] Step 1102 : The tablet computer detects the operating status of the target application.

[0123] Optionally, when the user performs a touch, slide, or other operation on the tablet computer, and determines that the target application on the tablet computer is in an operating state, step 1102 is executed again to continue detecting the operating state of the target application. If the target application on the tablet computer is in an inoperative state, step 1103 is executed.

[0124] In step 1103 , the tablet computer detects whether the specimen on the slide moves.

[0125] That is, when it is determined that the target application is not in operation, the tablet computer can continue to detect whether the specimen on the slide has moved. If so, the process returns to step 1102. If not, the process proceeds to step 1104. It should be noted that the detection order of steps 1102 and 1103 can also be reversed, which will not be repeated here.

[0126] Step 1104 : The tablet computer acquires a 1x magnified image of the target specimen.

[0127] Optionally, the process of the tablet computer acquiring the 1x target specimen image may refer to the description in 402 above, which will not be repeated here.

[0128] In step 1105 , the tablet computer performs specimen category recognition on the target specimen image to obtain the specimen category of the specimen.

[0129] In step 1106 , the tablet computer performs internal structure detection on the target specimen image to obtain the structural category and pixel coordinates of the constituent structures.

[0130] Optionally, the identification process from step 1105 to step 1106 may refer to the description in the above step 403 and will not be repeated here.

[0131] Step 1107 : Determine whether the image currently displayed on the display screen of the tablet computer corresponds to the image area position of the target specimen image.

[0132] Optionally, the implementation of the tablet computer in step 1107 may refer to the description in steps 404 to 406 above, which will not be repeated here.

[0133] Step 1108: Display the target recognition result on the display screen. The target recognition result includes the target display coordinates, the specimen category corresponding to the target display coordinates, and the structural category of the component structure at the target display coordinates. The target display coordinates are the pixel coordinates contained in the image area position.

[0134] Optionally, in step 1108 , the content of the target recognition result displayed by the tablet computer can refer to the description in the above steps 407 to 408 , which will not be repeated here.

[0135] In summary, the tablet computer of the present application obtains a 1x specimen image captured by a camera, performs image recognition on the 1x specimen image, and obtains an image recognition result; determines that the image currently displayed on the display screen corresponds to the image area position of the target specimen image, and displays the target recognition result on the display screen. The tablet computer can perform image recognition based on an image at a 1x magnification, and determine that the image currently displayed on the display screen corresponds to the image area position of the target specimen image, obtain the target recognition result corresponding to the image area position from the image recognition result and display it, and can recognize the captured target specimen image in real time, and display the recognition result of the image currently displayed on the display screen, so that the tablet computer can more conveniently display specimen information and improve the efficiency of the tablet computer in obtaining information about the target specimen image.

[0136] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0137] Please refer to Figure 12 , which shows a structural block diagram of an image recognition device provided by an exemplary embodiment of the present application. The image recognition device 1200 can be used Figure 1 In the scenario architecture shown, the terminal device in the scenario architecture is executed, and the terminal device includes a camera and a detachable slide assembly. The camera is facing the detachable slide assembly. The slide assembly includes a slide holder. The slide holder is used to place the slide to achieve Figure 2 , Figure 4 or Figure 11 The method provided in the embodiment shown is performed by the terminal device in whole or in part. The image recognition device 1200 may include the following modules:

[0138] An image acquisition module 1201 is configured to acquire a target specimen image captured by the camera, wherein the target specimen image is an image of the specimen contained on the slide at a basic magnification;

[0139] A result acquisition module 1202 is used to perform image recognition on the target specimen image to obtain an image recognition result;

[0140] A position determination module 1203 is used to determine the position of the image area of ​​the target specimen image corresponding to the image currently displayed on the display screen;

[0141] The result display module 1204 is configured to obtain a target recognition result corresponding to the position of the image area according to the image recognition result, and display the target recognition result on the display screen.

[0142] Optionally, the image recognition result includes the specimen category of the specimen, the structural category of the component structure of the specimen, and the pixel coordinates of the component structure in the target specimen image. The result acquisition module 1202 includes a first recognition unit and a second recognition unit.

[0143] The first recognition unit is configured to input the target specimen image into a first recognition model, recognize the target specimen image using the first recognition model, and obtain a specimen category corresponding to the target specimen image, wherein the first recognition model is trained based on a first sample image set, wherein the first sample image set includes a plurality of first specimen images at a basic magnification, each first specimen image carrying a corresponding specimen category label;

[0144] The second recognition unit is used to input the target specimen image into a second recognition model, recognize the target specimen image through the second recognition model, and obtain the structural category of the component structure contained in the target specimen image, as well as the pixel coordinates of the component structure in the target specimen image. The second recognition model is trained based on a second sample image set. The second sample image set includes multiple second specimen images at a basic magnification, and each second specimen image carries a structural category label of a corresponding component structure and a corresponding pixel coordinate label.

[0145] Optionally, the position determination module 1203 includes: a first determination unit, a coordinate conversion unit, and a second determination unit;

[0146] The first determining unit is configured to determine a magnification and movement information of a digital magnifying glass of the terminal device, wherein the movement information is a displacement of the target specimen image in the image coordinate system of the display screen after the target specimen image is magnified according to the magnification;

[0147] The coordinate conversion unit is configured to perform coordinate conversion on the pixel coordinates of the target specimen image according to the magnification and the movement information, so as to obtain the image coordinates of the target specimen image in the image coordinate system on the display screen;

[0148] The second determining unit is configured to determine a position of an image region in the target specimen image corresponding to the image currently displayed on the display screen according to the image coordinates in the image coordinate system.

[0149] Optionally, the result display module 1204 includes: a result selection unit and a display unit;

[0150] The result selection unit is configured to filter the target recognition result from the image recognition result based on the image area position and the image recognition result, wherein the target recognition result includes target display coordinates, a specimen category corresponding to the target display coordinates, and a structural category of a component structure at the target display coordinates, wherein the target display coordinates are pixel coordinates contained in the image area position;

[0151] The display unit is used to display the target recognition result in the display area of ​​the display screen according to a preset display method.

[0152] Optionally, the device further includes:

[0153] A magnification detection module, configured to detect whether the magnification of the digital magnifying glass of the terminal device is a basic magnification before the image acquisition module 1201 acquires the target specimen image captured by the camera;

[0154] a first acquisition module, configured to control the camera to photograph the glass slide placed on the glass slide holder when the magnification of the digital magnifying glass is the basic magnification, to obtain a first specimen image, and use the first specimen image as the target specimen image;

[0155] The second acquisition module is configured to acquire a second specimen image of a pre-stored basic magnification when the magnification of the digital magnifying glass is not the basic magnification, and use the second specimen image as the target specimen image.

[0156] Optionally, the device further includes:

[0157] A first acquisition module is configured to acquire, before the image acquisition module 1201 acquires the target specimen image captured by the camera, a current specimen image and N previous specimen images obtained by photographing the slide placed on the slide holder by the camera, where N is a positive integer;

[0158] A first calculation module is used to calculate the hash value of the current specimen image and the previous N frames of specimen images;

[0159] The first execution module is configured to execute the step of acquiring the target specimen image captured by the camera when the differences between the hash values ​​corresponding to the specimen images of adjacent frames are all less than a preset difference threshold.

[0160] Optionally, a target application is running in the terminal device, and an application interface corresponding to the target application includes an image of the specimen displayed on the display screen, and the apparatus further includes:

[0161] A second acquisition module is configured to acquire an operating state of the target application before the image acquisition module 1201 acquires the target specimen image captured by the camera, wherein the operating state includes an operating state and a non-operating state;

[0162] The second execution module is configured to execute the step of acquiring the target specimen image captured by the camera when the operation state of the target application is the non-operation state.

[0163] In summary, the terminal device of the present application obtains the target specimen image captured by the camera, and the target specimen image is the image of the specimen contained on the slide at the basic magnification; performs image recognition on the target specimen image to obtain the image recognition result; determines that the image currently displayed on the display screen corresponds to the image area position of the target specimen image; obtains the target recognition result corresponding to the image area position according to the image recognition result, and displays the target recognition result on the display screen. The terminal device can perform image recognition based on the image at the basic magnification, and determine that the image currently displayed on the display screen corresponds to the image area position of the target specimen image, obtain the target recognition result corresponding to the image area position from the image recognition result and display it, and can recognize the captured target specimen image in real time, and display the recognition result of the image currently displayed on the display screen, so that the terminal device can more conveniently display the specimen information, and improve the efficiency of the terminal device in obtaining the information of the target specimen image.

[0164] Please refer to Figure 13 , which shows a schematic diagram of the structure of a terminal device disclosed in an exemplary embodiment of the present application. Figure 13 As shown, it may include: Radio Frequency (RF) circuit 1310, memory 1320, input unit 1330, display unit 1340, sensor 1350, audio circuit 1360, WiFi module 1370, processor 1380, and power supply 1390. In the above embodiment, the terminal device can be used as a massage device or as a target device. Those skilled in the art will understand that Figure 13 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0165] The following combination Figure 13 This section introduces the various components of the terminal device:

[0166] The RF circuit 1310 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is sent to the processor 1380 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1310 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 1310 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0167] The memory 1320 can be used to store software programs and modules. The processor 1380 executes the various functional applications and data processing of the terminal device by running the software programs and modules stored in the memory 1320. The memory 1320 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the terminal device (such as audio data, a phone book, etc.). In addition, the memory 1320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0168] The input unit 1330 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the terminal device. Specifically, the input unit 1330 may include a touch panel 1331 and other input devices 1332. The touch panel 1331, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any other suitable object or accessory on or near the touch panel 1331) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1331 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch direction and detects the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 1380. It can also receive commands sent by the processor 1380 and execute them. In addition, the touch panel 1331 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1331, the input unit 1330 may further include other input devices 1332. Specifically, the other input devices 1332 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power keys, etc.), a trackball, a mouse, and a joystick.

[0169] The display unit 1340 can be used to display information input by the user or information provided to the user and various menus of the terminal device. The display unit 1340 may include a display panel 1341. Optionally, the display panel 1341 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1331 may cover the display panel 1341. When the touch panel 1331 detects a touch operation on or near it, it is transmitted to the processor 1380 to determine the type of touch event. Subsequently, the processor 1380 provides corresponding visual output on the display panel 1341 according to the type of touch event. Although in Figure 13 In the embodiment, the touch panel 1331 and the display panel 1341 are used as two independent components to realize the input and output functions of the terminal device, but in some embodiments, the touch panel 1331 and the display panel 1341 can be integrated to realize the input and output functions of the terminal device.

[0170] The terminal device may also include at least one sensor 1350, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1341 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1341 and / or the backlight when the terminal device is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the terminal device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the terminal device can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0171] Audio circuit 1360, speaker 1361, and microphone 1362 provide an audio interface between the user and the terminal device. Audio circuit 1350 converts received audio data into electrical signals and transmits them to speaker 1361, which then converts them into sound signals for output. Microphone 1362, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 1360 and converted into audio data. The audio data is then processed by processor 1380 and transmitted to, for example, another terminal device via RF circuit 1310, or the audio data is output to memory 1320 for further processing.

[0172] WiFi is a short-range wireless transmission technology. The terminal device can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1370. It provides users with wireless broadband Internet access. Figure 13 A WiFi module 1370 is shown, but it is understandable that it is not an essential component of the terminal device and can be omitted as needed without changing the essence of the invention.

[0173] Processor 1380 is the control center of the terminal device. It connects all components of the terminal device using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 1320 and accessing data stored in memory 1320, it performs various terminal device functions and processes data, thereby providing overall monitoring of the terminal device. Optionally, processor 1380 may include one or more processing units. Preferably, processor 1380 integrates an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1380.

[0174] The terminal device also includes a power supply 1390 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 1380 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0175] Although not shown, the terminal device may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0176] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the method in the above method embodiment when executed by a processor.

[0177] An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the method in the above method embodiment.

[0178] An embodiment of the present application discloses an application publishing platform, wherein the application publishing platform is used to publish a computer program product. When the computer program product runs on a computer, the computer executes the method in the above method embodiment.

[0179] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0180] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0181] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of this embodiment.

[0182] In addition, the functional units in the embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0183] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a memory and includes several requests for a computer device (which can be a personal computer, server or network device, etc., specifically a processor in a computer device) to execute some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0184] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0185] The above is an example introduction to an image recognition method, device, terminal device and storage medium disclosed in the embodiments of the present application. Individual examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image recognition method, characterized in that: The method is applied to a terminal device, the terminal device including a camera and a detachable glass slide assembly, the camera facing the detachable glass slide assembly, the glass slide assembly including a glass slide holder for placing a glass slide, and the terminal device also including a display screen, the method comprising: Acquiring a target specimen image captured by the camera, wherein the target specimen image is an image of the specimen contained on the slide at a basic magnification; Performing image recognition on the target specimen image to obtain an image recognition result; the image recognition result includes the specimen category of the specimen, the structural category of the component structure of the specimen, and the pixel coordinates of the component structure in the target specimen image; Determining that an image currently displayed on the display screen corresponds to an image region position of the target specimen image; the image currently displayed on the display screen is a full or partial image of the target specimen image displayed after a magnification or movement operation, and the image region position is a coordinate region position in a pixel coordinate system of the target specimen image; Filtering a target recognition result from the image recognition result according to the image area position and the image recognition result, the target recognition result including target display coordinates, a specimen category corresponding to the target display coordinates, and a structural category of a component structure at the target display coordinates, wherein the target display coordinates are pixel coordinates contained in the image area position; The target recognition result is displayed in the display area of ​​the display screen according to a preset display method.

2. The method according to claim 1, characterized in that The performing image recognition on the target specimen image to obtain an image recognition result includes: Inputting the target specimen image into a first recognition model, recognizing the target specimen image using the first recognition model to obtain a specimen category corresponding to the target specimen image, wherein the first recognition model is trained based on a first sample image set, wherein the first sample image set includes a plurality of first specimen images at a basic magnification, each first specimen image carrying a corresponding specimen category label; The target specimen image is input into a second recognition model, and the target specimen image is recognized by the second recognition model to obtain a structural category of a component structure contained in the target specimen image and a pixel coordinate of the component structure in the target specimen image. The second recognition model is trained based on a second sample image set, and the second sample image set includes multiple second specimen images at a basic magnification, each second specimen image carrying a structural category label of a corresponding component structure and a corresponding pixel coordinate label.

3. The method according to claim 2, characterized in that The step of determining the position of the image currently displayed on the display screen in the image area of ​​the target specimen image includes: Determining the magnification and movement information of the digital magnifying glass of the terminal device, wherein the movement information is the displacement of the target specimen image in the image coordinate system of the display screen after the target specimen image is magnified according to the magnification; performing coordinate conversion on pixel coordinates of the target specimen image according to the magnification and the movement information to obtain image coordinates of the target specimen image in an image coordinate system on the display screen; The position of the image region corresponding to the image currently displayed on the display screen in the target specimen image is determined according to the image coordinates in the image coordinate system.

4. The method according to claim 1, wherein Before acquiring the target specimen image captured by the camera, the method further includes: Detecting whether the magnification of the digital magnifying glass of the terminal device is a basic magnification; When the magnification of the digital magnifying glass is the basic magnification, controlling the camera to photograph the glass slide placed on the glass slide holder to obtain a first specimen image, and using the first specimen image as the target specimen image; When the magnification of the digital magnifying glass is not the basic magnification, a second specimen image of a pre-stored basic magnification is acquired, and the second specimen image is used as the target specimen image.

5. The method according to any one of claims 1 to 4, characterized in that: Before acquiring the target specimen image captured by the camera, the method further includes: Acquire a current specimen image and previous N frames of specimen images obtained by photographing the glass slide placed on the glass slide holder with the camera, where N is a positive integer; Calculating hash values ​​of the current specimen image and the previous N frames of specimen images; When the differences between the hash values ​​corresponding to the specimen images of adjacent frames are all less than a preset difference threshold, the step of acquiring the target specimen image captured by the camera is performed.

6. The method according to any one of claims 1 to 4, characterized in that: A target application is running in the terminal device, and an application interface corresponding to the target application includes an image of the specimen displayed on the display screen. Before acquiring the target specimen image captured by the camera, the method further includes: Acquire the operating state of the target application, where the operating state includes an operating state and a non-operating state; When the operation state of the target application is the non-operation state, the step of acquiring the target specimen image captured by the camera is performed.

7. An image recognition device, characterized in that: The device is applied to a terminal device, the terminal device including a camera and a detachable glass slide assembly, the camera facing the detachable glass slide assembly, the glass slide assembly including a glass slide holder, the glass slide holder for placing a glass slide, the terminal device also including a display screen, and the device including: An image acquisition module, configured to acquire a target specimen image captured by the camera, wherein the target specimen image is an image of the specimen contained on the slide at a basic magnification; a result acquisition module, configured to perform image recognition on the target specimen image to obtain an image recognition result; the image recognition result includes the specimen category of the specimen, the structural category of the component structure of the specimen, and the pixel coordinates of the component structure in the target specimen image; a position determination module, configured to determine an image region position corresponding to an image currently displayed on the display screen in the target specimen image; the image currently displayed on the display screen is a full or partial image of the target specimen image displayed after a magnification or movement operation, and the image region position is a coordinate region position in a pixel coordinate system of the target specimen image; The result display module includes: a result selection unit and a display unit; The result selection unit is configured to filter out a target recognition result from the image recognition result based on the image area position and the image recognition result, the target recognition result including target display coordinates, a specimen category corresponding to the target display coordinates, and a structural category of a component structure at the target display coordinates, wherein the target display coordinates are pixel coordinates contained in the image area position; The display unit is used to display the target recognition result in the display area of ​​the display screen according to a preset display method.

8. A terminal device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the image recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image recognition method according to any one of claims 1 to 6 is implemented.

Images