Object recognition method and device, electronic device and medium

Abstract

The present disclosure provides an object recognition method, including: determining a first mapping relationship between a visible light image and a dermoscopic image of a first object, wherein the visible light image is an image collected using a first lens, and the dermoscopic image does not need to be collected, but if the dermoscopic image is collected, it needs to be collected using a second lens, and the magnification of the second lens is greater than the magnification of the first lens; based on the visible light image of the first object, determining corresponding visible light image feature information; and based on the determined first mapping relationship and the visible light image feature information, recognizing the first object. The present disclosure also provides an object recognition device, an electronic device, and a computer-readable storage medium.

Description

Technical Field

[0001] The present disclosure relates to an object recognition method, an object recognition device, an electronic device, and a computer-readable storage medium. Background Art

[0002] When using computer vision technology to identify skin diseases, image data of three scale modalities are usually used: visible light image data, dermoscopic image data, and pathological image data.

[0003] Currently, in the doctor's workflow, visible light images are first used to identify lesions. If the visible light images cannot be used for identification, dermoscopic images are further used for lesion identification. If the dermoscopic images cannot be used for identification, pathological images are further used for lesion identification.

[0004] This is because, among the image data of these three scale modalities, visible light image data is universal, and visible light images have the advantages of non-invasive collection and low price. However, due to the insufficient amount of information, visible light image data has the disadvantage of low accuracy when identifying lesions for certain diseases. Although dermoscopic images also have the advantage of non-invasive collection, they are expensive and require professional equipment for collection, which many hospitals do not have the conditions to achieve. The collection of pathological images will damage the skin, and has the disadvantages of being expensive and having a long waiting period.

[0005] Therefore, using visible light images for lesion identification for certain diseases is currently the most economical and time-saving method. However, due to the uncertainty of visible light images for lesion identification of certain diseases, visible light images need to be combined with high-scale modality images to improve accuracy. Summary of the invention

[0006] One aspect of the present disclosure provides an object recognition method, comprising: determining a first mapping relationship between a visible light image and a dermoscopic image of a first object, wherein the visible light image is an image captured using a first lens, and the dermoscopic image does not need to be captured, but if the dermoscopic image is captured, it needs to be captured using a second lens, and the magnification of the second lens is greater than the magnification of the first lens; based on the visible light image of the first object, determining corresponding visible light image feature information; and recognizing the first object based on the determined first mapping relationship and the visible light image feature information.

[0007] Optionally, determining a first mapping relationship between a visible light image and a dermoscopic image for a first object includes: determining visible light image data of the visible light image; inputting the visible light image data into a first encoder to obtain first feature information output by an intermediate layer of the first encoder, wherein the first encoder is capable of encoding the visible light image data of a target object to obtain dermoscopic image data for the target object, and the target object includes the first object; and characterizing the first mapping relationship using the first feature information.

[0008] Optionally, the method also includes: training the first encoder; the training of the first encoder includes: acquiring visible light images and dermoscopic images for multiple second objects, wherein the visible light images for the multiple second objects are images captured using the first lens, and the dermoscopic images for the multiple second objects are images captured using the second lens; and training the first encoder based on the visible light images and dermoscopic images for the multiple second objects.

[0009] Optionally, identifying the first object based on the determined first mapping relationship and the visible light image feature information includes: determining a first feature vector based on the first mapping relationship and the visible light image feature information; inputting the first feature vector into a first classification model to output a corresponding classification result; and identifying the first object based on the classification result.

[0010] Optionally, the method also includes: training the first classification model; the training of the first classification model includes: acquiring visible light images and dermoscopic images for multiple third objects, wherein the visible light images for the multiple third objects are images captured using the first lens, and the dermoscopic images for the multiple third objects are images captured using the second lens; determining multiple third mapping relationships between the visible light images and the dermoscopic images for the multiple third objects; and training the first classification model based on the visible light images for the multiple third objects and the multiple third mapping relationships.

[0011] Another aspect of the present disclosure provides an object recognition device, including: a first determination module, used to determine a first mapping relationship between a visible light image and a dermoscopic image for a first object, wherein the visible light image is an image captured using a first lens, and the dermoscopic image does not need to be captured, but if the dermoscopic image is captured, it is necessary to use a second lens for capture, and the magnification of the second lens is greater than the magnification of the first lens; a second determination module, used to determine corresponding visible light image feature information based on the visible light image for the first object; and an identification module, used to identify the first object based on the determined first mapping relationship and the visible light image feature information.

[0012] Optionally, the first determination module includes: a determination unit, used to determine the visible light image data of the visible light image; a first acquisition unit, used to input the visible light image data into a first encoder to obtain first feature information output by an intermediate layer of the first encoder, wherein the first encoder is capable of encoding the visible light image data of a target object to obtain dermoscopic image data for the target object, and the target object includes the first object; and a processing unit, used to characterize the first mapping relationship using the first feature information.

[0013] Optionally, the device also includes: a training module for training the first encoder; the training module includes: a second acquisition unit for acquiring visible light images and dermoscopic images for multiple second objects, wherein the visible light images for the multiple second objects are images captured using the first lens, and the dermoscopic images for the multiple second objects are images captured using the second lens; and a training unit for training the first encoder based on the visible light images and dermoscopic images for the multiple second objects.

[0014] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method of the embodiment of the present disclosure.

[0015] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method of the embodiment of the present disclosure when executed.

[0016] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement the above method of the embodiment of the present disclosure when being executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] For a more complete understanding of the present disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1 The application scenarios of the object recognition method and device according to the embodiments of the present disclosure are schematically shown;

[0019] Figure 2 A flowchart of an object recognition method according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 3A A schematic diagram of a depth encoder according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 3B A schematic diagram schematically shows a schematic diagram of extracting feature information from a visible light image according to an embodiment of the present disclosure;

[0022] Figure 3C A schematic diagram schematically shows object recognition based on a visible light image according to an embodiment of the present disclosure;

[0023] Figure 4 A block diagram schematically shows an object recognition device according to an embodiment of the present disclosure; and

[0024] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0028] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some boxes or combinations thereof in the block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions can create a device for implementing the functions / operations described in these block diagrams and / or flow charts when executed by the processor. The technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable storage medium storing instructions, which can be used by an instruction execution system or used in conjunction with an instruction execution system.

[0030] The embodiments of the present disclosure provide an object recognition method that does not require the collection of dermoscopic images but can achieve the recognition accuracy that can be achieved by lesion recognition based on dermoscopic images, and an object recognition device that can apply the method. The method includes the following operations. Determine a first mapping relationship between a visible light image and a dermoscopic image of a first object, wherein the visible light image is an image collected using a first lens, and the dermoscopic image does not need to be collected, but if the dermoscopic image is collected, it is necessary to use a second lens for collection, and the magnification of the second lens is greater than the magnification of the first lens. Based on the visible light image of the first object, determine the corresponding visible light image feature information. Based on the determined first mapping relationship and the visible light image feature information, identify the first object.

[0031] Figure 1 The application scenarios of the object recognition method and device according to the embodiments of the present disclosure are schematically shown. It should be noted that: Figure 1 The examples shown are merely scenarios in which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0032] Currently, in the doctor's workflow, the first step is to use Figure 1 The visible light image 101 shown is used to identify skin diseases. If the visible light image 101 cannot be used for identification, the dermoscopic image 102 will be further used for skin disease identification. If the dermoscopic image 102 still cannot be used for identification, the pathological image 103 will be further used for skin disease identification.

[0033] It can be seen that in the prior art, only image data of one scale modality, such as visible light image data, can be used to identify skin diseases. However, since the amount of information contained in visible light images is insufficient, only using visible light image data, which is one scale modality, to identify skin diseases results in poor accuracy.

[0034] In order to overcome the problem of poor accuracy when using only one scale modality of image data, such as visible light image data, to identify skin diseases, a solution using a multi-scale modality fusion model to identify lesions is also provided in the related art. However, this solution requires training a multi-scale modality fusion model.

[0035] Since both dermoscopic images and pathological images have the shortcomings described above and are not universal, and complete image data of the three scale modalities only occur in patients with very serious conditions, this method obviously has too high requirements on data integrity and does not meet clinical needs.

[0036] In this regard, the disclosed embodiment provides an improved object recognition method. This method only requires the acquisition of a patient's visible light image, without further acquiring a dermoscopic image of the patient, but can achieve the same recognition accuracy based on a dermoscopic image based only on a visible light image.

[0037] The present disclosure will be described in detail below with reference to specific embodiments.

[0038] Figure 2 The flowchart of the object recognition method according to the embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the method may include, for example, operations S210 to S230.

[0040] In operation S210, a first mapping relationship between a visible light image and a dermoscopic image for a first object is determined, wherein the visible light image is an image captured using a first lens, and the dermoscopic image does not need to be captured. However, if the dermoscopic image is to be captured, it is necessary to use a second lens for capture, and the magnification of the second lens is greater than the magnification of the first lens.

[0041] It should be noted that in the embodiment of the present disclosure, the visible light image can be collected using the first device, wherein the optical lens used by the first device for collection is the first lens. The dermoscopic image does not need to be specially collected in the embodiment of the present disclosure, but if such an image is collected, it is necessary to use a special second device for collection, wherein the optical lens used by the second device for collection is the second lens. The magnification of the second lens is greater than the magnification of the first lens.

[0042] Specifically, in order to understand the skin condition of any part of the skin of any person, in the embodiment of the present disclosure, only the visible light image of the corresponding skin can be collected for this person, without collecting the dermoscopic image of the skin. Because the information output by the encoding part (i.e., the encoder part) of the depth encoder from the visible light image to the dermoscopic image contains the high-order semantic information common to the visible light image and the corresponding dermoscopic image, after collecting the visible light image of a certain part of the skin of a person, the visible light image can be input into the depth encoder, and the corresponding feature information can be extracted from the information output by the encoding part of the depth encoder, and then the extracted feature information is used as the mapping relationship between the visible light image and the dermoscopic image of the skin of the person.

[0043] Figure 3A A depth encoder according to an embodiment of the present disclosure is schematically shown.

[0044] like Figure 3A As shown, for the depth encoder 31, its left end is the visible light image input layer 311, its right end is the dermoscopic image output layer 312, and the information output by the middle layer 313 is the information output by the encoding part of the depth encoder.

[0045] Since the disclosed embodiment only needs to collect images of small-scale modalities, that is, only needs to collect visible light images, it has the advantages of non-invasive collection and low price. Moreover, collecting visible light images does not require particularly professional equipment, so many hospitals have the conditions to implement it.

[0046] It should be noted that, in the disclosed embodiment, the first object may be a pigmented nevus, melanoma, red birthmark, hemangioma, etc., grown on any part of the skin of any person. In the disclosed embodiment, pigmented nevus, melanoma, red birthmark, hemangioma, etc. may be collectively referred to as lesions.

[0047] In operation S220, corresponding visible light image feature information is determined based on the visible light image for the first object.

[0048] It should be noted that in the embodiments of the present disclosure, any image feature extraction method may be used to extract corresponding image features from a visible light image, and this is not limited herein.

[0049] For example, Figure 3B A schematic diagram of extracting feature information from a visible light image according to an embodiment of the present disclosure is schematically shown. Figure 3B As shown, a visible light image A can be input into the neural network 32 for convolution operation and sampling processing, and the information output by the neural network 32 is used as the feature information of the visible light image.

[0050] In operation S230, the first object is recognized based on the determined first mapping relationship and the visible light image feature information.

[0051] Specifically, the feature information of the visible light image and the mapping information used to characterize the mapping relationship determined based on the visible light image may be fused (e.g., spliced) to obtain fused feature information, and the fused feature information is input into a pre-trained classifier for classification, and then the lesion of the first object is identified based on the classification result. For example, it is determined based on the classification result whether the first object is a nevus or a melanoma.

[0052] Through the embodiments of the present disclosure, since only visible light images of small-scale modalities need to be collected, it has the advantages of non-invasive collection and low price, and collecting visible light images does not require particularly professional equipment, so many hospitals have the conditions to implement it.

[0053] Furthermore, through the embodiments of the present disclosure, in addition to considering the image features of the visible light image, the mapping relationship between the dermoscopic image corresponding to the visible light image (that is, the dermoscopic image and the aforementioned visible light image are for the same object) is also considered during the object recognition process, that is, the characteristic information of the dermoscopic image corresponding to the visible light image is also considered at the same time (equivalent to using image data of two scale modalities through an image of one scale modality). Therefore, compared with the related art that can only use image data of one scale modality through an image of one scale modality, the embodiments of the present disclosure have more reference information and higher recognition accuracy.

[0054] In addition, although the embodiments of the present disclosure involve the fusion of image data of multiple scale modalities, it does not require the collection of images of multiple scale modalities. For example, only visible light images need to be collected, and dermoscopic images do not need to be collected. It is not necessary to have complete image data of multiple scale modalities. Obviously, this method does not have high requirements on data integrity and meets clinical needs.

[0055] As an optional embodiment, determining a first mapping relationship between a visible light image and a dermoscopic image of a first object may include, for example, the following operations.

[0056] Visible light image data of the visible light image is determined.

[0057] The visible light image data is input into a first encoder to obtain first feature information output by an intermediate layer of the first encoder, wherein the first encoder is capable of encoding the visible light image data of a target object to obtain dermoscopic image data for the target object, and the target object includes a first object.

[0058] The first mapping relationship is characterized by using the first feature information.

[0059] Specifically, return reference Figure 3A In order to obtain a mapping relationship between a visible light image and its corresponding dermoscopic image, for example, the visible light image can be input into the visible light image input end 311 of the depth encoder 31 (such as the first encoder), and output information can be obtained from the middle layer 313 of the depth encoder 31, and then the output information is used to characterize the mapping relationship between the visible light image and its corresponding dermoscopic image.

[0060] Since the information output by the middle layer of the deep encoder from the visible light image to the corresponding dermoscopic image contains high-order semantic information common to the visible light image and the corresponding dermoscopic image, after collecting a visible light image of a certain part of a person's skin, the visible light image can be input into the deep encoder, and the corresponding feature information can be quickly extracted from the information output by the encoding part of the depth encoder, and used as the mapping relationship between the visible light image of the person's skin at that part and the corresponding dermoscopic image.

[0061] Further, as an optional embodiment, the method may also include, for example, training the first encoder.

[0062] Training the first encoder may include, for example, the following operations.

[0063] Visible light images and dermoscopic images for the multiple second objects are acquired, wherein the visible light images for the multiple second objects are images acquired using the first lens, and the dermoscopic images for the multiple second objects are images acquired using the second lens.

[0064] The first encoder is trained based on the visible light images and the dermoscopic images for a plurality of second objects.

[0065] In the embodiment of the present disclosure, the visible light images and dermoscopic images of previous patients can be collected as training samples of the embodiment of the present disclosure. It should be noted that in the training samples, the visible light images and dermoscopic images need to appear in pairs, and the paired visible light images and dermoscopic images must be collected from the same lesion of the same patient. In addition, in the embodiment of the present disclosure, both the visible light images and the dermoscopic images need to be actually collected.

[0066] Specifically, after obtaining the training samples, the neural network can be used to train the deep encoder from the visible light image to the dermoscopic image to learn the mapping relationship from the visible light image to the dermoscopic image. After obtaining the trained deep encoder, the visible light image is directly input into the input end of the deep encoder, and the information output by the encoder is taken out. This information integrates the high-level semantic information shared by the visible light image and the corresponding dermoscopic image. Return to reference Figure 3A , where the network on the left of the middle layer 313 in the figure is the encoder part, and the network on the right of the middle layer 313 in the figure is the decoder part.

[0067] It should be noted here that when predicting the mapping relationship from the visible light image to the dermoscopic image, since only the output information of the encoder part (i.e., the information output by the middle layer 313) is used, it is not necessary to collect the corresponding dermoscopic image, thereby saving the high cost of collecting the dermoscopic image.

[0068] As an optional embodiment, identifying the first object based on the determined first mapping relationship and the visible light image feature information may include the following operations, for example.

[0069] A first feature vector is determined based on the first mapping relationship and the visible light image feature information.

[0070] The first feature vector is input into the first classification model to output a corresponding classification result.

[0071] Based on the classification result, the first object is identified.

[0072] Specifically, the feature information representing the first mapping relationship may be fused (eg, spliced) with the feature information of the corresponding visible light image to determine the corresponding first feature vector.

[0073] For example, assuming that a visible light image A of a patient's back skin black spot is collected, according to the above-mentioned embodiments of the present disclosure, the mapping relationship between the visible light image A and its corresponding dermoscopic image A can be predicted from the visible light image A, where the feature information X 1 To characterize the mapping relationship, the feature vector F 1 (H, W, T 1 ) indicates that T 1 For example, 512-dimensional features may be included. In addition, feature information X can be obtained by extracting image features from the visible light image A. 2 , using the eigenvector F 2 (H, W, T 2 ), T 2 For example, it can include 1024-dimensional features. When performing lesion identification, F 1(H, W, T 1 ) and F 2 (H, W, T 2 ) to form F 3 (H, W, T 3 ), where T 3 =(T 1 +T 2 ), T 3 For example, it can include T 1 Included 512-dimensional features and T 2 The 1024-dimensional features included are 1536-dimensional features in total, and then F 3 (H, W, T 3 ) is used as the first feature vector to input the pre-trained first classification model so as to determine whether the black spots on the patient's back skin are nevus or melanoma.

[0074] Figure 3C A schematic diagram of object recognition based on a visible light image according to an embodiment of the present disclosure is schematically shown.

[0075] like Figure 3C As shown, the above-mentioned visible light image A can be input into the depth encoder 31 in the figure, and feature information X can be extracted from the information output by the middle layer 313 (not shown in the figure) of the depth encoder 31. 1 At the same time, the above-mentioned visible light image A can be input into the neural network 32 in the figure to output the feature information X of the visible light image A. 2 Finally, the feature information X 1 The corresponding eigenvector F 1 (H, W, T 1 ) and the characteristic information X 2 The corresponding eigenvector F 2 (H, W, T 2 ) The fusion F 3 (H, W, T 3 ) is input into the first classification model 33 in the figure, and the output information of the first classification model 33 (i.e., the classification result) is obtained, and then the black spot on the back skin of the patient is determined to be a pigmented nevus or a melanoma according to the information output by the first classification model 33. In addition, Figure 3C As shown, the neural network 32 includes a convolution layer 321 and a pooling layer 322, wherein the convolution layer 321 is used to implement convolution operations, and the pooling layer 322 is used to implement sampling operations.

[0076] Further, as an optional embodiment, the method may also include, for example, training a first classification model.

[0077] Training the first classification model may include, for example, the following operations.

[0078] Visible light images and dermoscopic images for multiple third objects are acquired, wherein the visible light images for the multiple third objects are images acquired using the first lens, and the dermoscopic images for the multiple third objects are images acquired using the second lens.

[0079] A plurality of third mapping relationships between the visible light image and the dermoscopic image for a plurality of third objects are determined.

[0080] A first classification model is trained based on the visible light images of the plurality of third objects and the plurality of third mapping relationships.

[0081] In the embodiment of the present disclosure, similar to the aforementioned embodiment of training the first encoder, specifically, multiple visible light images and dermoscopic images of the third object can be used as samples, and the samples can be divided into training samples and prediction samples. According to the method provided in the aforementioned embodiment of training the first encoder, the training samples can be used to train the corresponding deep encoder, and then the prediction samples can be input into the deep encoder to predict the mapping relationship between each visible light image in the prediction sample and its corresponding dermoscopic image. At the same time, the feature information of each visible light image in the prediction sample is extracted. Then, the feature information of each visible light image in the prediction sample is fused with the feature information used to characterize the mapping relationship corresponding to each visible light image to obtain the fused feature information. Finally, the fused feature information is used to train the first classification model in the embodiment of the present disclosure.

[0082] Through the embodiments of the present disclosure, when training a visible light classification model (i.e., the first classification model), feature information output by the encoder part of each visible light image in the prediction sample via the deep encoder is fused with feature information extracted from each visible light image in the prediction sample, and training is performed based on the fused feature information. In this way, feature information of related dermoscopic images learned through the deep encoder can be added to the visible light classification network, thereby improving the accuracy of the visible light classification model.

[0083] In other words, through the embodiments of the present disclosure, when using visible light images for classification and identification, the information contained in the dermoscopic image is added through the mapping relationship, thereby improving the accuracy of identification. And during training, the feature information in the dermoscopic image is encoded into the visible light classification network through the mapping relationship, so that only visible light images need to be collected during testing, which is in line with the doctor's workflow.

[0084] Figure 4 The block diagram of the object recognition device according to the embodiment of the present disclosure is schematically shown.

[0085] like Figure 4As shown, the object recognition device 400 includes a first determination module 410, a second determination module 420 and a recognition module 430. The object recognition device 400 can execute the above reference Figure 2 The method described is to achieve the recognition effect achieved by image recognition based on dermoscopic images without taking dermoscopic images.

[0086] Specifically, the first determination module 410 is used to determine a first mapping relationship between a visible light image and a dermoscopic image for a first object, wherein the visible light image is an image captured using a first lens, and the dermoscopic image does not need to be captured, but if the dermoscopic image is captured, it is necessary to use a second lens for capture, and the magnification of the second lens is greater than the magnification of the first lens.

[0087] The second determination module 420 is configured to determine corresponding visible light image feature information based on the visible light image of the first object.

[0088] The recognition module 430 is used to recognize the first object based on the determined first mapping relationship and the visible light image feature information.

[0089] Through the embodiments of the present disclosure, since only visible light images of small-scale modalities need to be collected, it has the advantages of non-invasive collection and low price, and collecting visible light images does not require particularly professional equipment, so many hospitals have the conditions to implement it.

[0090] Furthermore, through the embodiments of the present disclosure, in addition to considering the image features of the visible light image, the mapping relationship between the dermoscopic image corresponding to the visible light image (that is, the dermoscopic image and the aforementioned visible light image are for the same object) is also considered during the object recognition process, that is, the characteristic information of the dermoscopic image corresponding to the visible light image is also considered at the same time (equivalent to using image data of two scale modalities through an image of one scale modality). Therefore, compared with the related art that can only use image data of one scale modality through an image of one scale modality, the embodiments of the present disclosure have more reference information and higher recognition accuracy.

[0091] In addition, although the embodiments of the present disclosure involve the fusion of image data of multiple scale modalities, it does not require the collection of images of multiple scale modalities. For example, only visible light images need to be collected, and dermoscopic images do not need to be collected. It is not necessary to have complete image data of multiple scale modalities. Obviously, this method does not have high requirements on data integrity and meets clinical needs.

[0092] As an optional embodiment, the first determination module includes: a determination unit, configured to determine visible light image data of a visible light image. A first acquisition unit, configured to input the visible light image data into a first encoder to obtain first feature information output by an intermediate layer of the first encoder, wherein the first encoder can encode the visible light image data of a target object to obtain dermoscopic image data of the target object, and the target object includes a first object. A processing unit, configured to characterize a first mapping relationship using the first feature information.

[0093] Further, as an optional embodiment, the device further includes: a training module for training the first encoder. The training module includes: a second acquisition unit for acquiring visible light images and dermoscopic images for multiple second objects, wherein the visible light images for the multiple second objects are images collected using the first lens, and the dermoscopic images for the multiple second objects are images collected using the second lens. The training unit is used to train the first encoder based on the visible light images and dermoscopic images for the multiple second objects.

[0094] It should be noted that, in the embodiments of the present disclosure, the embodiments of the apparatus part are identical or similar to the embodiments of the method part, and will not be described in detail here.

[0095] According to the modules and units of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and units of the embodiments of the present disclosure, any one or more can be split into multiple modules for implementation. According to the modules and units of the embodiments of the present disclosure, any one or more can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and units of the embodiments of the present disclosure, one or more can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0096] For example, any multiple of the first determination module 410, the second determination module 420, and the identification module 430 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first determination module 410, the second determination module 420, and the identification module 430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in any appropriate combination of any of them. Alternatively, at least one of the first determination module 410, the second determination module 420, and the identification module 430 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0097] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0098] like Figure 5 As shown, the electronic device 500 includes a processor 510 and a computer-readable storage medium 520. The electronic device 500 can execute the method according to the embodiment of the present disclosure.

[0099] Specifically, the processor 510 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 510 may also include an onboard memory for cache purposes. The processor 510 may be a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0100] The computer-readable storage medium 520 may be, for example, a non-volatile computer-readable storage medium, and specific examples include but are not limited to: a magnetic storage device, such as a magnetic tape or a hard disk (HDD); an optical storage device, such as a compact disk (CD-ROM); a memory, such as a random access memory (RAM) or flash memory; and the like.

[0101] The computer-readable storage medium 520 may include a computer program 521 , which may include code / computer-executable instructions that, when executed by the processor 510 , cause the processor 510 to perform a method according to an embodiment of the present disclosure or any variation thereof.

[0102] The computer program 521 may be configured to have, for example, a computer program code including a computer program module. For example, in an example embodiment, the code in the computer program 521 may include one or more program modules, for example, 521A, module 521B, ... It should be noted that the division method and number of modules are not fixed, and those skilled in the art may use appropriate program modules or program module combinations according to actual conditions, and when these program module combinations are executed by the processor 510, the processor 510 may execute the method according to the embodiment of the present disclosure or any variation thereof.

[0103] According to an embodiment of the present disclosure, at least one of the first determination module 410, the second determination module 420, and the identification module 430 may be implemented as a reference. Figure 5 The computer program modules described herein, when executed by the processor 510, can implement the corresponding operations described above.

[0104] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0105] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0106] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0108] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A method for object recognition, comprising: determining visible light image data for a visible light image of a first object; The visible light image data is input into a first encoder to obtain first feature information output by an intermediate layer of the first encoder; wherein the first feature information represents a first mapping relationship between the visible light image and the dermoscopic image, the visible light image is an image captured by a first lens, and the dermoscopic image does not need to be captured, but if the dermoscopic image is captured, a second lens needs to be used for the capture, and the magnification of the second lens is greater than the magnification of the first lens; the first encoder can encode the visible light image data of a target object to obtain dermoscopic image data of the target object, and the target object includes the first object; Based on the visible light image for the first object, determining corresponding visible light image feature information; and The first object is identified based on the determined first mapping relationship and the visible light image feature information.

2. The method according to claim 1, wherein: The method further comprises: training the first encoder; The training of the first encoder comprises: Acquire visible light images and dermoscopic images for a plurality of second objects, wherein the visible light images for the plurality of second objects are images acquired using the first lens, and the dermoscopic images for the plurality of second objects are images acquired using the second lens; and The first encoder is trained based on the visible light images and dermoscopic images for the plurality of second objects.

3. The method according to claim 1, wherein: The identifying the first object based on the determined first mapping relationship and the visible light image feature information includes: Determine a first feature vector based on the first mapping relationship and the visible light image feature information; Inputting the first feature vector into a first classification model to output a corresponding classification result; and Based on the classification result, the first object is identified.

4. The method according to claim 3, wherein: The method further comprises: training the first classification model; The training of the first classification model comprises: Acquire visible light images and dermoscopic images for a plurality of third objects, wherein the visible light images for the plurality of third objects are images acquired using the first lens, and the dermoscopic images for the plurality of third objects are images acquired using the second lens; determining a plurality of third mapping relationships between the visible light images and the dermoscopic images for the plurality of third objects; and The first classification model is trained based on the visible light images for the plurality of third objects and the plurality of third mapping relationships.

5. An object recognition device, comprising: A first determining module, configured to determine visible light image data of a visible light image of a first object; The visible light image data is input into a first encoder to obtain first feature information output by an intermediate layer of the first encoder; wherein the first feature information represents a first mapping relationship between the visible light image and the dermoscopic image, the visible light image is an image captured by a first lens, and the dermoscopic image does not need to be captured, but if the dermoscopic image is captured, a second lens needs to be used for the capture, and the magnification of the second lens is greater than the magnification of the first lens; the first encoder can encode the visible light image data of a target object to obtain dermoscopic image data of the target object, and the target object includes the first object; a second determining module, configured to determine corresponding visible light image feature information based on the visible light image for the first object; and A recognition module is used to recognize the first object based on the determined first mapping relationship and the visible light image feature information.

6. The device according to claim 5, wherein: The apparatus further comprises: a training module, configured to train the first encoder; The training module includes: a second acquisition unit, configured to acquire visible light images and dermoscopic images for a plurality of second objects, wherein the visible light images for the plurality of second objects are images acquired using the first lens, and the dermoscopic images for the plurality of second objects are images acquired using the second lens; and A training unit is used to train the first encoder based on the visible light images and dermoscopic images for multiple second objects.

7. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

8. A computer-readable storage medium storing computer-executable instructions, wherein the instructions are used to implement the method according to any one of claims 1 to 4 when executed.

Images