Similarity determination, network training, search method and device and storage medium

By mapping text and image data to the same semantic space, using neural networks to calculate the similarity between patients, the problem of low similarity calculation accuracy in the prior art is solved, and the performance of similar object search and personalized medical services are improved. quality.

CN111091010BActive Publication Date: 2024-12-03BOE TECHNOLOGY GROUP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201911159185.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-11-22
Publication Date
2024-12-03
Estimated Expiration
2040-08-05

AI Technical Summary

Technical Problem

When calculating the similarity between patients, the prior art cannot effectively utilize the information complementarity of different modal data and the verification and review characteristics of each other, resulting in low accuracy in the calculation of similarity and poor search performance.

Method used

By mapping data of different modalities (such as text and images) to the same semantic comparison space, neural networks use to map data subitems to semantic representations, and calculate similarity based on these representations to improve the accuracy of data similarity calculation.

Benefits of technology

It improves the accuracy of data similarity calculation and the performance of similar object search, and can find patients similar to a given patient more accurately, improving the personalization of medical services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111091010B_ABST
    Figure CN111091010B_ABST
Patent Text Reader

Abstract

A method and device for determining data similarity, a non-transitory storage medium, a method and device for training a neural network, and a method for searching similar objects. The method for determining data similarity comprises: obtaining first data of a first object, wherein the first data comprises first sub-data of a first modality or a second modality; mapping the first sub-data to a first semantic representation in a semantic comparison space, wherein the semantic comparison space makes it possible to calculate similarity between the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space; obtaining second data of a second object, wherein the second data comprises second sub-data of the first modality or the second modality; mapping the second sub-data to a second semantic representation in the semantic comparison space; and calculating the similarity between the first data and the second data based at least on the first semantic representation and the second semantic representation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data similarity determination method, a data similarity determination device, a non-transitory storage medium, a neural network training method and device for data similarity determination, and a similar object search method and device. Background Art

[0002] With the rapid development of medical informatization, electronic medical data is growing exponentially. The massive amount of electronic medical data provides opportunities for big data analysis and mining. For example, for a given patient, patients similar to the given patient can be found from a large amount of electronic medical data. The diagnosis, treatment, and rehabilitation information of these patients similar to the given patient can be used as an important reference in the following aspects: assessing health risks, selecting treatment plans, estimating hospital stays and treatment costs, thereby providing personalized medical services for given patients and improving the quality of medical services. The above process of providing personalized medical services using information about similar patients is similar to the process of doctors making clinical decisions after observing a large amount of clinical case data, and is of great clinical significance. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a method for determining data similarity, which includes: obtaining first data of a first object, wherein the first data includes first sub-data of a first modality or a second modality; mapping the first sub-data to a first semantic representation in a semantic comparison space, wherein the semantic comparison space is such that a semantic representation obtained by mapping the data of the first modality to the semantic comparison space and a semantic representation obtained by mapping the data of the second modality to the semantic comparison space are calculable similarity; obtaining second data of a second object, wherein the second data includes second sub-data of the first modality or the second modality; mapping the second sub-data to a second semantic representation in the semantic comparison space; and calculating the similarity between the first data and the second data based at least on the first semantic representation and the second semantic representation.

[0004] For example, in at least one example of the data similarity determination method, the first modality is text and the second modality is an image.

[0005] For example, in at least one example of the method for determining data similarity, the first data includes first sub-data for the first modality and third sub-data for the second modality; the second data includes second sub-data for the first modality and fourth sub-data for the second modality; the first object includes a first characteristic, the first sub-data includes a first sub-semantic describing the first characteristic, and the third sub-data includes a third sub-semantic describing the first characteristic; the second object includes a second characteristic, the second sub-data includes a second sub-semantic describing the second characteristic, and the fourth sub-data includes a fourth sub-semantic describing the second characteristic; the method also includes: mapping the third sub-data to a third semantic representation in the semantic comparison space, and mapping the fourth sub-data to a fourth semantic representation in the semantic comparison space; the calculating the similarity between the first data and the second data based at least on the first semantic representation and the second semantic representation includes: obtaining the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation.

[0006] For example, in at least one example of the data similarity determination method, obtaining the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation includes: calculating at least one of the similarity between the first semantic representation and the fourth semantic representation and the similarity between the second semantic representation and the third semantic representation.

[0007] For example, in at least one example of the data similarity determination method, obtaining the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation also includes: calculating at least one of the similarity between the first semantic representation and the second semantic representation and the similarity between the third semantic representation and the fourth semantic representation.

[0008] For example, in at least one example of the data similarity determination method, the similarity between the first data and the second data is equal to the sum of the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation.

[0009] For example, in at least one example of the data similarity determination method, the first semantic representation, the second semantic representation, the third semantic representation, and the fourth semantic representation are all expressed as vectors; and the dimension of the vector corresponding to the first semantic representation, the dimension of the vector corresponding to the second semantic representation, the dimension of the vector corresponding to the third semantic representation, and the dimension of the vector corresponding to the fourth semantic representation are equal to each other.

[0010] For example, in at least one example of the method for determining data similarity, the similarity f(p i ,p j ) is obtained using the following expression:

[0011] f(p i ,p j )=f((t i ,g i ),(t j ,g j ))=sim1(t i ,t j )+sim2(t i ,g j )+sim2(g i ,t j )+sim1(g i ,g j ),

[0012] rt i =NN1(t i ),rg i =NN2(g i ),rt j =NN1(t j ),rg j =NN2(g j ),

[0013]

[0014]

[0015]

[0016]

[0017] Among them, p i is the first data, p j is the second data, t i is the first sub-data, g i is the third sub-data, t j is the second sub-data, gj is the fourth sub-data, sim1(t i ,t j ) is the similarity between the first sub-data and the second sub-data, sim1(g i ,g j ) is the similarity between the third sub-data and the fourth sub-data, sim2(t i ,g j ) is the similarity between the first sub-data and the fourth sub-data, sim2(g i ,t j ) is the similarity between the second sub-data and the third sub-data, NN1 is the mapping of the corresponding sub-data using the first neural network, NN2 is the mapping of the corresponding sub-data using the second neural network, rt i For the first semantic representation, rg i is the third semantic representation, rt j For the second semantic representation, rg j is the fourth semantic representation.

[0018] For example, in at least one example of the data similarity determination method, a first neural network is used to map the first sub-data to the first semantic representation and the second sub-data to the second semantic representation; and a second neural network is used to map the third sub-data to the third semantic representation and the fourth sub-data to the fourth semantic representation.

[0019] At least one embodiment of the present disclosure also provides a data similarity determination device, which includes: an acquisition module, a mapping module and a similarity calculation module. The acquisition module is configured to acquire first data of a first object and second data of a second object, wherein the first data includes first sub-data of a first modality or a second modality, and the second data includes second sub-data of the first modality or the second modality; the mapping module is configured to map the first sub-data to a first semantic representation in a semantic comparison space, and map the second sub-data to a second semantic representation in the semantic comparison space, wherein the semantic comparison space makes it possible to calculate the similarity of the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space; and the similarity calculation module is configured to calculate the similarity between the first data and the second data based at least on the first semantic representation and the second semantic representation.

[0020] For example, in at least one example of the data similarity determination device, the first data includes first sub-data for the first modality and third sub-data for the second modality; the second data includes second sub-data for the first modality and fourth sub-data for the second modality; the first object includes a first characteristic, the first sub-data includes a first sub-semantic describing the first characteristic, and the third sub-data includes a third sub-semantic describing the first characteristic; the second object includes a second characteristic, the second sub-data includes a second sub-semantic describing the second characteristic, and the fourth sub-data includes a fourth sub-semantic describing the second characteristic; the mapping module is also configured to map the third sub-data to a third semantic representation in the semantic comparison space, and to map the fourth sub-data to a fourth semantic representation in the semantic comparison space; the similarity calculation module is configured to obtain the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation, and the fourth semantic representation.

[0021] For example, in at least one example of the data similarity determination device, the similarity calculation module is configured to: calculate the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation. The similarity between the first data and the second data is equal to the sum of the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation.

[0022] At least one embodiment of the present disclosure further provides a data similarity determination device, which includes: a processor and a memory, wherein the memory stores computer program instructions suitable for execution by the processor, and when the computer program instructions are executed by the processor, the processor executes any data similarity determination method provided by at least one embodiment of the present disclosure.

[0023] At least one embodiment of the present disclosure further provides a non-transitory storage medium, which includes computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the computer executes any data similarity determination method provided by at least one embodiment of the present disclosure.

[0024] At least one embodiment of the present disclosure also provides a training method for a neural network for determining data similarity, comprising: acquiring first training data of a training object, wherein the first training data comprises first training sub-data of a first modality and second training sub-data of a second modality, the training object comprises training characteristics, the first training sub-data comprises first training sub-semantics describing the training characteristics, and the second training sub-data comprises second training sub-semantics describing the training characteristics; mapping the first training sub-data into a first training semantic representation in a semantic comparison space through a first neural network, wherein the semantic comparison space is such that the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space are calculable similarity; mapping the second training sub-data into a second training semantic representation in the semantic comparison space through a second neural network; calculating the similarity between the first training semantic representation and the second training semantic representation, and adjusting the parameters of at least one of the first neural network and the second neural network according to the similarity between the first training semantic representation and the second training semantic representation.

[0025] For example, in at least one example of the training method, adjusting the parameters of at least one of the first neural network and the second neural network according to the similarity between the first training semantic representation and the second training semantic representation includes: calculating a loss function based on the similarity between the first training semantic representation and the second training semantic representation; and minimizing the loss function by adjusting the parameters of at least one of the first neural network and the second neural network.

[0026] For example, in at least one example of the training method, the loss function l is obtained using the following expression:

[0027]

[0028] Among them, trt i is the first training sub-data of the i-th training object, trg i is the second training sub-data of the i-th training object, sim2(trt i ,trg i ) is the similarity between the first training sub-data of the i-th training object and the second training sub-data of the i-th training object, NN1 is the mapping of the corresponding training sub-data using the first neural network, NN2 is the mapping of the corresponding training sub-data using the second neural network, NN1(trt i ) is the first training semantic representation of the i-th training object, NN2(trg i) is the second training semantic representation of the i-th training object; and minimizing the loss function by adjusting the parameters of at least one of the first neural network and the second neural network includes adjusting the parameters of at least one of the first neural network and the second neural network based on a gradient descent method.

[0029] At least one embodiment of the present disclosure further provides a training device for a neural network for determining data similarity, comprising: a processor and a memory. The memory stores computer program instructions suitable for execution by the processor, and when the computer program instructions are executed by the processor, the processor executes any training method provided by at least one embodiment of the present disclosure.

[0030] At least one embodiment of the present disclosure also provides a similar object search method, which includes: obtaining first data of a first object; obtaining second data of multiple second objects; using any data similarity determination method provided by at least one embodiment of the present disclosure to calculate the similarity between the first object and each of the second objects to obtain multiple object similarities; and outputting information of the second object corresponding to the object similarity with the largest value among the multiple object similarities.

[0031] For example, in at least one example of the similar object search method, the similar object search method further includes: sorting the plurality of second objects based on the plurality of object similarities.

[0032] At least one embodiment of the present disclosure further provides a similar object search device, which includes: any data similarity determination device provided by at least one embodiment of the present disclosure.

[0033] For example, in at least one example of the similar object search device, the similar object search device further comprises an object database. The object database stores data of multiple objects and provides at least part of the object data to the data similarity determination device as second data of multiple second objects.

[0034] For example, in at least one example of the similar object search device, the similar object search device further includes a human-computer interaction interface. The human-computer interaction interface is configured to receive first data of a first object and provide the first data to the data similarity determination device; the data similarity determination device is configured to provide the similarity between the first object and each of the second objects as multiple object similarities; the human-computer interaction interface is further configured to output information of the second object corresponding to the object similarity with the largest value among the multiple object similarities.

[0035] For example, in at least one example of the similar object search device, the similar object search device further comprises an information sending receiver. The information sending receiver is configured to receive first data of a first object from a client and provide the first data to the data similarity determination device; the data similarity determination device is configured to provide the similarity between the first object and each of the second objects as multiple object similarities; the information sending receiver is further configured to provide information of the second object corresponding to the object similarity with the largest value among the multiple object similarities to the client.

[0036] At least one embodiment of the present disclosure also provides a similar object search device, which includes: a human-computer interaction interface and an information transmitter and receiver. The human-computer interaction interface is configured to receive first data of a first object and provide the first data to the information transmitter and receiver; the information transmitter and receiver is configured to provide the first data to a server; the information transmitter and receiver is also configured to receive second data of a second object having the greatest similarity to the first data of the first object from the server, and provide the second data of the second object to the human-computer interaction interface; the similarity between the first object and each of the second objects is determined by any data similarity determination device provided by at least one embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0038] Figure 1A A schematic diagram of a convolutional neural network is shown.

[0039] Figure 1B It is a schematic diagram of the structure of a convolutional neural network;

[0040] Figure 1C It is a schematic diagram of the working process of a convolutional neural network;

[0041] Figure 2 is a flow chart of a method for determining data similarity provided by at least one embodiment of the present disclosure;

[0042] Figure 3A yes Figure 2 An example of a method for determining data similarity is shown;

[0043] Figure 3B yes Figure 2 Another example of the method for determining data similarity shown;

[0044] Figure 4is an exemplary block diagram of a data similarity determination device provided by at least one embodiment of the present disclosure;

[0045] Figure 5 is an exemplary block diagram of another data similarity determination device provided by at least one embodiment of the present disclosure;

[0046] Figure 6 is an exemplary block diagram of a non-transitory storage medium provided by at least one embodiment of the present disclosure;

[0047] Figure 7 is a flowchart of a neural network training method for determining data similarity provided by at least one embodiment of the present disclosure;

[0048] Figure 8 is an exemplary block diagram of a neural network training apparatus for determining data similarity provided by at least one embodiment of the present disclosure;

[0049] Fig. 9 is a flowchart of a similar object search method provided by at least one embodiment of the present disclosure;

[0050] Fig.10 is an exemplary block diagram of a similar object search device provided by at least one embodiment of the present disclosure;

[0051] Fig.11 yes Fig.10 An example of a similar object search device is shown;

[0052] Fig.12 yes Fig.10 Another example of the similar object finding device shown;

[0053] Fig.13 is an exemplary block diagram of another similar object searching device provided by at least one embodiment of the present disclosure;

[0054] Fig.14 A similar object search system provided by at least one embodiment of the present disclosure;

[0055] Fig.15 is an exemplary scene graph of a similar object search system provided by at least one embodiment of the present disclosure; and

[0056] Fig.16 A computing device provided by at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0058] Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with ordinary skills in the field to which the present disclosure belongs. "First", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, "include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0059] Artificial Intelligence (AI) involves theories, methods, technologies and application systems that use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain results. Among the algorithm technologies in the field of AI, deep learning has received widespread attention from academia and industry. As deep learning has made breakthrough progress in areas such as image classification, object detection and natural language processing, the demand for applying it to real-life scenarios has become increasingly strong.

[0060] Convolutional Neural Network (CNN) can be used to recognize two-dimensional shapes. CNN mainly simplifies the complexity of neural network models and reduces the number of weights through local receptive fields and weight sharing. With the development of deep learning technology, the application scope of CNN is no longer limited to the field of image recognition. It can also be used in face recognition, text recognition, animal classification, image processing and other fields.

[0061] Figure 1A A schematic diagram of a convolutional neural network is shown. For example, the convolutional neural network can be used for image processing, which uses images as input and output and replaces scalar weights with convolution kernels. Figure 1AOnly a convolutional neural network with a three-layer structure is shown in FIG. , and the embodiments of the present disclosure are not limited to this. Figure 1A As shown, the convolutional neural network includes an input layer 101, a hidden layer 102, and an output layer 103. The input layer 101 has 4 inputs, the hidden layer 102 has 3 outputs, and the output layer 103 has 2 outputs. Finally, the convolutional neural network outputs 2 images.

[0062] For example, the four inputs of the input layer 101 may be four images, or four feature images of one image. The three outputs of the hidden layer 102 may be feature images of the image input through the input layer 101 .

[0063] For example, Figure 1A As shown, the convolutional layer has weights and bias Weight Represents the convolution kernel, bias is a scalar added to the output of the convolutional layer, where k is the label representing the input layer 101, and i and j are the labels of the units of the input layer 101 and the units of the hidden layer 102, respectively. For example, the first convolutional layer 201 includes a first set of convolution kernels ( Figure 1A In ) and the first set of biases ( Figure 1A In The second convolutional layer 202 includes a second set of convolutional kernels ( Figure 1A In ) and the second set of biases ( Figure 1A In ). Typically, each convolutional layer includes dozens or hundreds of convolution kernels. If the convolutional neural network is a deep convolutional neural network, it may include at least five convolutional layers.

[0064] For example, Figure 1AAs shown, the convolutional neural network also includes a first activation layer 203 and a second activation layer 204. The first activation layer 203 is located after the first convolutional layer 201, and the second activation layer 204 is located after the second convolutional layer 202. The activation layer (for example, the first activation layer 203 and the second activation layer 204) includes an activation function, and the activation function is used to introduce nonlinear factors to the convolutional neural network so that the convolutional neural network can better solve more complex problems. The activation function may include a rectified linear unit (ReLU) function, a sigmoid function, or a hyperbolic tangent function (tanh function), etc. The ReLU function is an unsaturated nonlinear function, and the Sigmoid function and the tanh function are saturated nonlinear functions. For example, the activation layer can be used alone as a layer of the convolutional neural network, or the activation layer can also be included in the convolutional layer (for example, the first convolutional layer 201 can include the first activation layer 203, and the second convolutional layer 202 can include the second activation layer 204).

[0065] For example, in the first convolutional layer 201, first, a number of convolution kernels in the first set of convolution kernels are applied to each input. and some of the biases in the first set Then, the output of the first convolution layer 201 can be processed by the first activation layer 203 to obtain the output of the first activation layer 203. In the second convolution layer 202, first, a plurality of convolution kernels in the second set of convolution kernels are applied to the output of the first activation layer 203. and some of the biases in the second set of biases Then, the output of the second convolution layer 202 can be processed by the second activation layer 204 to obtain the output of the second activation layer 204. For example, the output of the first convolution layer 201 can be the convolution kernel applied to its input. Then with bias As a result of the addition, the output of the second convolution layer 202 can be the convolution kernel applied to the output of the first activation layer 203. Then with bias The result of addition.

[0066] Before using a convolutional neural network for image processing, the convolutional neural network needs to be trained. After training, the convolution kernel and bias of the convolutional neural network remain unchanged during image processing. During the training process, each convolution kernel and bias are adjusted through multiple sets of input / output example images and optimization algorithms to obtain an optimized convolutional neural network model.

[0067] Figure 1B It is a schematic diagram of the structure of a convolutional neural network; Figure 1C This is a schematic diagram of the working process of a convolutional neural network. For example, Figure 1B As shown, the convolutional neural network includes a first layer, a second layer, a third layer, a fully connected layer, etc.

[0068] For example, Figure 1B and Figure 1C As shown in Figure 2, after the input image is input into the convolutional neural network through the input layer, it undergoes several processing steps in sequence (for example, Figure 1B The first level, second level, and third level in the output are the category identifiers.

[0069] It should be noted that although Figure 1B Only the convolutional neural network is shown to include three levels (i.e., the first level, the second level, and the third level), but the embodiments of the present disclosure are not limited thereto. For example, the convolutional neural network may also include two levels, four levels, or other applicable numbers of levels.

[0070] For example, Figure 1B and Figure 1C As shown, each of the first level, the second level, and the third level may include a convolution module and a downsampling layer, that is, the main components of the convolutional neural network may include multiple convolution layers, multiple downsampling layers, and fully connected layers. For example, the processing process of each level may include: convolution (convolution) and downsampling (sub-sampling / down-sampling) of the input image.

[0071] The convolution layer is the core layer of the convolutional neural network. In the convolution layer of the convolutional neural network, a neuron is only connected to some neurons in the adjacent layers. The convolution layer can apply several convolution kernels (also called filters) to the input image to extract multiple types of features of the input image. Each convolution kernel can extract one type of feature. The convolution kernel is generally initialized in the form of a random decimal matrix. During the training process of the convolutional neural network, the convolution kernel will be learned to obtain reasonable weights. The result obtained after applying a convolution kernel to the input image is called a feature image (feature map), and the number of feature images is equal to the number of convolution kernels. Each feature image is composed of some rectangularly arranged neurons, and the neurons of the same feature image share weights. The shared weights here are the convolution kernels. The feature image output by a convolution layer of one level can be input to the convolution layer of the next adjacent level and processed again to obtain a new feature image. For example, Figure 1B As shown, the convolution layer of the first level can output a first feature image, and the first feature image is input to the convolution layer of the second level and processed again to obtain a second feature image.

[0072] For example, Figure 1CAs shown in the figure, the convolution layer can use different convolution kernels to convolve the data of a local receptive field of the input image. The convolution result is input into the activation layer, which calculates according to the corresponding activation function to obtain the feature information of the input image.

[0073] For example, Figure 1B and Figure 1C As shown in the figure, the downsampling layer is set between adjacent convolutional layers, and the downsampling layer is a form of downsampling. On the one hand, the downsampling layer can be used to reduce the scale of the input image, simplify the complexity of calculation, and reduce the phenomenon of overfitting to a certain extent; on the other hand, the downsampling layer can also perform feature compression and extract the main features of the input image. The downsampling layer can reduce the size of the feature image but does not change the number of feature images. For example, an input image of size 12×12 is sampled by a 6×6 convolution kernel, and then a 2×2 output image can be obtained, which means that 36 pixels on the input image are merged into 1 pixel in the output image. The last downsampling layer or convolutional layer of the convolutional neural network can be connected to one or more fully connected layers, which are used to connect all the extracted features. The output of the fully connected layer is a one-dimensional matrix, that is, a vector.

[0074] In addition, the most exciting breakthrough that deep learning has brought to natural language processing is word embedding. The word embedding technology converts words into vectors, that is, maps words to a semantic space to obtain vectors. In this semantic space, similar words have similar word vectors, so the similarity between different words can be calculated. For example, neural networks can be used to map words to vector space. For example, in natural language processing applications, word embedding is used as the input feature of deep learning models.

[0075] For example, word vector technology can be divided into word vector technology based on statistical methods and word vector technology based on language models. For example, word vector technology based on statistical methods can be divided into word vector technology based on co-occurrence matrix and word vector technology based on singular value decomposition. For example, in word vector technology based on language model, the language model generates word vectors by training the neural network language model NNLM (neural network language model), and the word vector is a by-product of the language model. For example, word vector technology based on language model includes word vector technology based on word2vec. For example, word2vec is implemented by borrowing the method of neural network, using skip-gram model and continuous bag of words model (CBOW). For example, the skip-gram model uses a word as input to predict the context around it, and the CBOW model uses the context of a word as input to predict the word itself. For example, the training method of the skip-gram model and the continuous bag of words model can be found in the relevant technology, which will not be repeated here.

[0076] In addition, a recurrent neural network (RNN) model can also be used to analyze word vectors, and the obtained word vectors are input into the RNN for analysis. The long short-term memory model is a special recurrent neural network (RNN) model, which is proposed to solve the problem of gradient diffusion in the RNN model; in traditional RNN, the training algorithm uses the back propagation through time (BPTT) algorithm. When the time is long, the residual error to be transmitted back will decrease exponentially, resulting in slow update of network weights, which cannot reflect the effect of long-term memory of RNN. Therefore, a storage unit is needed to store memory. In response to this, researchers proposed the LSTM model. The idea of ​​the long short-term memory model is to replace each hidden unit in the RNN with a cell with a memory function, that is, to replace the small circle of the hidden layer of the RNN network with a block of LSTM. For example, the long short-term memory model includes a forget gate (used to determine which information continues to pass through the unit), an input gate (used to determine how much new information is added to the unit) and an output gate (used to determine the output). For example, the specific structure and working principle of the long short-term memory network can be found in the relevant technology, which will not be repeated here.

[0077] The inventor of the present disclosure has noticed that when performing comprehensive analysis of different types of data, the key problem of similar object search is how to calculate (eg, accurately calculate) the similarity between two objects. The following is an example of similar patient search in medical-related data analysis.

[0078] The key problem of similar patient search is how to obtain the similarity between two patients. One method of obtaining the similarity between two patients is to calculate the similarity between the single modality data of the two patients. For example, the similarity between the two patients is obtained by calculating the similarity between the text information recorded in the electronic medical records of the two patients; for another example, the similarity between the two patients' imaging data (for example, computer tomography (CT) images, X-ray scan images, magnetic resonance images, etc.) is calculated to obtain the similarity between the two patients.

[0079] The inventors of the present disclosure also noticed that, in theory, the electronic medical record data and image data generated by the same patient in the same visit have semantic consistency. For example, the following description is on the medical record of a patient: "A nodule with a clear boundary can be seen in the left upper lung, and the size of the nodule is 2.2 cm × 1.4 cm"; the patient's chest PET-CT (positron emission tomography-computerized X-ray tomography) image also shows a nodule in the left upper lung. At this time, the medical record data and the image data describe the same objective fact, that is, they have semantic consistency. However, since the electronic medical record data and image data generated by the same patient in the same visit use different modalities and methods to describe the patient's condition (i.e., using language and images to describe, respectively), the electronic medical record data and image data generated by the same patient in the same visit have information complementarity. In addition, the electronic medical record data and image data can also verify and review the information provided by each other.

[0080] The inventors of the present disclosure have also noticed that since the relevant patient similarity calculation method obtains the similarity between two patients by calculating the similarity between the data of a single modality of the two patients, the patient similarity calculation method cannot utilize the information complementarity of different modality data and / or the characteristics of different modality data to verify and check each other, so the accuracy of the patient similarity calculation result is low. Correspondingly, the search performance of the similar patient search method based on the above patient similarity calculation method may be poor (for example, the similar patients found may not be the patients who are most similar to the given patient).

[0081] At least one embodiment of the present disclosure provides a method for determining data similarity, a device for determining data similarity, a non-transitory storage medium, a training method and device for a neural network for determining data similarity, and a method for searching similar objects. The method for determining data similarity includes: obtaining first data of a first object, wherein the first data includes first sub-data of a first modality or a second modality; mapping the first sub-data to a first semantic representation in a semantic comparison space, wherein the semantic comparison space makes it possible to calculate the similarity of the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space; obtaining second data of a second object, wherein the second data includes second sub-data of the first modality or the second modality; mapping the second sub-data to a second semantic representation in the semantic comparison space; and calculating the similarity between the first data and the second data based on at least the first semantic representation and the second semantic representation. The method for determining data similarity improves the accuracy of the calculation result of data similarity. For example, the semantic comparison space can be a semantic space in which semantic representations can be compared to calculate the similarity between the semantic representations.

[0082] The following non-restrictive description is given through several examples and embodiments of the data similarity determination method, data similarity determination device, non-temporary storage medium, neural network training method and device for data similarity determination, and similar object search method and device provided according to the embodiments of the present disclosure. As described below, different features in these specific examples and embodiments can be combined with each other without conflicting with each other to obtain new examples and embodiments, and these new examples and embodiments also fall within the scope of protection of the present disclosure.

[0083] It should be noted that, for the sake of clarity, the data similarity determination method provided by at least one embodiment of the present disclosure is exemplified below using patient data as an example, but the data similarity determination method provided by at least one embodiment of the present disclosure is not limited to the similarity applied to patient data. For example, the data similarity determination method provided by at least one embodiment of the present disclosure can also be applied to other scenarios involving the use of different types of data to describe the same or similar characteristics, such as the similarity of mechanical equipment flaw detection data (including ultrasonic scanning data, optical image data, etc.). Correspondingly, the similar object search method based on the data similarity determination method provided by at least one embodiment of the present disclosure can be applied not only to the search for similar patients, but also to the search for similar objects in other scenarios.

[0084] Figure 2 is a flow chart of a method for determining data similarity provided by at least one embodiment of the present disclosure. Figure 2 The method for determining data similarity includes the following steps S110 to S150.

[0085] Step S110: Acquire first data of a first object. For example, the first data includes first sub-data of a first modality or a second modality.

[0086] Step S120: Map the first sub-data to a first semantic representation in a semantic comparison space. For example, the semantic comparison space allows the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space to be similar.

[0087] Step S130: Acquire second data of the second object. For example, the second data includes second sub-data of the first modality or the second modality.

[0088] Step S140: Map the second sub-data into a second semantic representation in the semantic comparison space.

[0089] Step S150: Calculate the similarity between the first data and the second data based on at least the first semantic representation and the second semantic representation.

[0090] For example, by mapping the first data of the first object and the second data of the second object into the same semantic space, that is, the semantic comparison space of the embodiment of the present disclosure, that is, in a semantic comparison space in which the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space are calculable similarities, the information complementarity of data of different modalities can be utilized (for example, by utilizing multimodal fusion), or data of different modalities can be used to verify and review each other (for example, data of images can be used to verify and review data of medical records), so that the data similarity determination method provided by at least one embodiment of the present disclosure can improve the accuracy of the data similarity calculation results.

[0091] For example, the data similarity determination method can be performed in the order of step S110, step S120, step S130, step S140, and step S150, but the embodiments of the present disclosure are not limited thereto. For another example, the data similarity determination method can be performed in the order of step S110+step S130 (that is, step S110 and step S130 are performed simultaneously), step S120+step S140 (that is, step S120 and step S140 are performed simultaneously), and step S150.

[0092] For example, the first object may be a given patient, that is, a patient currently seeking medical treatment; the second object may be a reference patient to be determined to be a similar patient to the given patient, that is, a reference patient whose similarity with the given patient needs to be determined.

[0093] For example, the first modality is different from the second modality. For example, the first modality is text, and the second modality is an image (e.g., an image). For example, the text includes at least one of words and numbers. For example, the text includes text in a medical record and at least one text related to physiological characteristics provided by a wearable terminal. For example, the text in a medical record may describe the patient's basic information, symptoms, diagnosis results, treatment plans, etc. For example, the image includes at least one of an image related to physiological characteristics provided by a medical imaging device and a wearable terminal. For example, the image may include an image showing human physiological information such as a computer X-ray tomography image, a nuclear magnetic resonance image, an ultrasound image, an electrocardiogram, an electroencephalogram, optical photography, etc. The image may include a static or dynamic image. For example, a video image may be an image including multiple frames of images.

[0094] For example, a neural network may be used to map the first sub-data to a first semantic representation in a semantic comparison space, and a neural network may be used to map the second sub-data to a second semantic representation in a semantic comparison space. For example, in a semantic comparison space (e.g., an n-dimensional space), both the first semantic representation and the second semantic representation are expressed as vectors (correspondingly, n-dimensional vectors), and the vectors have the same dimension, so that the similarity between the first semantic representation and the second semantic representation can be calculated based on the value of the vector in each dimension.

[0095] For example, the semantic comparison space may be an n-dimensional Euclidean space, in which the inner product and distance between vectors may be calculated, so that the semantic representation obtained by mapping the data of the first modality to the Euclidean space and the semantic representation obtained by mapping the data of the second modality to the Euclidean space can be calculated similarity. For example, a neural network may map the data of the first modality and the data of the second modality with a high similarity to adjacent areas of the Euclidean space, and make the vector obtained by mapping the data of the first modality and the vector obtained by mapping the data of the second modality have the same dimension.

[0096] In one example, the first data includes first sub-data for a first modality and third sub-data for a second modality; the second data includes second sub-data for the first modality and fourth sub-data for the second modality; the first object includes a first characteristic, the first sub-data includes a first sub-semantic describing the first characteristic, and the third sub-data includes a third sub-semantic describing the first characteristic; the second object includes a second characteristic, the second sub-data includes a second sub-semantic describing the second characteristic, and the fourth sub-data includes a fourth sub-semantic describing the second characteristic.

[0097] In the above example, the data similarity determination method further includes: mapping the third sub-data into a third semantic representation in the semantic comparison space, and mapping the fourth sub-data into a fourth semantic representation in the semantic comparison space.

[0098] In the above example, calculating the similarity between the first data and the second data based on at least the first semantic representation and the second semantic representation includes: obtaining the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation.

[0099] Figure 3A yes Figure 2 An example of a method for determining data similarity is shown in FIG. Figure 3A As shown, the data similarity determination method includes the following steps S210 to S230.

[0100] Step S210: Acquire first data of a first object and second data of a second object.

[0101] For example, the first object may be a given patient, that is, a patient currently seeking medical treatment; the second object may be a reference patient whose similarity to the given patient needs to be determined.

[0102] For example, the first data includes first sub-data of the first modality and third sub-data of the second modality; the second data includes second sub-data of the first modality and fourth sub-data of the second modality.

[0103] For example, the first modality is text and the second modality is image; in this case, the first data includes first sub-data which is text and third sub-data which is image; the second data includes second sub-data which is text and fourth sub-data which is image.

[0104] For example, a first object includes a first characteristic, a first sub-data includes a first sub-semantic describing the first characteristic, and a third sub-data includes a third sub-semantic describing the first characteristic; a second object includes a second characteristic, a second sub-data includes a second sub-semantic describing the second characteristic, and a fourth sub-data includes a fourth sub-semantic describing the second characteristic. For example, the first characteristic and the second characteristic are both features that describe the same aspect of the object. For example, the first characteristic describes a characteristic related to the lungs of a given patient, and the second characteristic describes a characteristic related to the lungs of a reference patient.

[0105] For example, the first sub-semantics and the third sub-semantics describe the first characteristic based on the first modality and the second modality, respectively, and the second sub-semantics and the fourth sub-semantics describe the second characteristic based on the use of the first modality and the second modality. For example, the first sub-semantics of the first sub-data include the following description "a nodule with a clear boundary can be seen in the left upper lung, and the size of the nodule is 2.2 cm × 1.4 cm"; the third sub-semantics of the third sub-data include the part of the chest PET-CT (positron emission computed tomography-computerized X-ray tomography technology) image showing a nodule in the left upper lung.

[0106] Step S220: Map the first sub-data to the first semantic representation in the semantic comparison space, map the second sub-data to the second semantic representation in the semantic comparison space, map the third sub-data to the third semantic representation in the semantic comparison space, and map the fourth sub-data to the fourth semantic representation in the semantic comparison space.

[0107] For example, the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation are all expressed as vectors, that is, the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation are all vectors in the semantic comparison space and have the same dimension, so the similarity between different semantic representations can be calculated based on the size of each dimension.

[0108] For example, the first semantics representing the dimension of the corresponding vector, the second semantics representing the dimension of the corresponding vector, the third semantics representing the dimension of the corresponding vector, and the fourth semantics representing the dimension of the corresponding vector can be preset according to actual experience.

[0109] For example, the dimension of the vector corresponding to the first semantic representation, the dimension of the vector corresponding to the second semantic representation, the dimension of the vector corresponding to the third semantic representation, and the dimension of the vector corresponding to the fourth semantic representation are equal to each other. For example, by making the dimension of the vector corresponding to the first semantic representation, the dimension of the vector corresponding to the second semantic representation, the dimension of the vector corresponding to the third semantic representation, and the dimension of the vector corresponding to the fourth semantic representation equal to each other, any two of the first semantic representation, the second semantic representation, the third semantic representation, and the fourth semantic representation can be similarity-calculated.

[0110] For example, the first sub-semantics, the second sub-semantics, the third sub-semantics and the fourth sub-semantics correspond to the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation, respectively. For example, mapping the first sub-data to the first semantic representation in the semantic comparison space, mapping the second sub-data to the second semantic representation in the semantic comparison space, mapping the third sub-data to the third semantic representation in the semantic comparison space, and mapping the fourth sub-data to the fourth semantic representation in the semantic comparison space includes: mapping the first sub-semantics to the first semantic representation in the semantic comparison space, mapping the second sub-semantics to the second semantic representation in the semantic comparison space, mapping the third sub-semantics to the third semantic representation in the semantic comparison space, and mapping the fourth sub-semantics to the fourth semantic representation in the semantic comparison space.

[0111] For example, the semantic comparison space allows the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space to be calculable similarity, that is, the semantic comparison space allows the data of different modalities to be calculable similarity in the semantic comparison space, thereby Figure 3A The data similarity determination method shown can utilize the information complementarity of data of different modalities and / or the characteristics of data of different modalities to verify and check each other, thereby improving the accuracy of the data similarity calculation result.

[0112] For example, mapping the first sub-data, the second sub-data, the third sub-data and the fourth sub-data into the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation of the semantic comparison space respectively can reduce the dimension of at least part of the data, thereby reducing the amount of computation required for calculating similarity.

[0113] In one example, the first sub-data is mapped to the first semantic representation of the semantic comparison space using the first neural network and the second sub-data is mapped to the second semantic representation of the semantic comparison space using the second neural network, the third sub-data is mapped to the third semantic representation of the semantic comparison space using the second neural network and the fourth sub-data is mapped to the fourth semantic representation of the semantic comparison space; that is, the sub-data of the first modality is mapped to the semantic comparison space using the first neural network and the sub-data of the second modality is mapped to the semantic comparison space using the second neural network. For example, the first neural network is different from the second neural network so as to map sub-data of different modalities to the same semantic comparison space; the first neural network and the second neural network are trained neural networks (for example, neural networks trained by machine learning).

[0114] For example, the first semantic representation rt i , the third semantic representation rg i , the second semantic representation rt j , the fourth semantic representation rg jIt can be obtained using the following expression:

[0115] rt i =NN1(t i ),rg i =NN2(g i ),rt j =NN1(t j ),rg j =NN2(g j ),

[0116] Here, t i is the first sub-data, g i is the third sub-data, t j is the second sub-data, g j is the fourth sub-data, NN1 uses the first neural network to map the corresponding sub-data, and NN2 uses the second neural network to map the corresponding sub-data.

[0117] For example, the trained first neural network can decompose the description of the medical record "A nodule with a relatively clear boundary can be seen in the left upper lung, and the size of the nodule is 2.2 cm × 1.4 cm" into: the organ type is lung (the corresponding number is 9), the organ has an abnormality (the corresponding number is 2), the abnormality is a nodule (the corresponding number is 4), the location of the abnormality is the upper left (the corresponding number is 1), and the size of the abnormality is 2.2 cm × 1.4 cm. In this case, the trained first neural network can convert the content describing the lung-related characteristics in the medical record into a first vector (9, 2, 4, 1, 2.2, 1.4). For example, the method of converting text into a vector by a neural network (e.g., a long short-term memory network neural network) can be found in the relevant technology and will not be repeated here.

[0118] For example, the trained second neural network has the function of confirming the organ type and whether the organ is abnormal in the medical image; and, if there is an abnormality, the trained second neural network also has the function of determining the type, location and size of the abnormality. For example, the trained second neural network can confirm that the organ type in the chest PET-CT image is the lung (the corresponding number is 9), the organ is abnormal (the corresponding number is 2), the abnormality is a nodule (the corresponding number is 4), the position of the abnormality in the image is the upper left (the corresponding number is 1), and the size of the abnormality is 2.2 cm × 1.4 cm. In this case, the trained second neural network can convert the information provided by the area describing the lung-related characteristics in the chest PET-CT image into a second vector (9, 2, 4, 1, 2.2, 1.4). For example, since the first vector and the second vector are the same, the similarity between the above medical record and the medical image can be obtained as 1. For example, the method for a neural network (e.g., a convolutional neural network) to identify the type, size and position of an object in an image can be referred to the relevant technology, which will not be repeated here.

[0119] For example, in order to make sub-data of different modalities maintain semantic consistency after being mapped to the semantic comparison space, the positions of semantically consistent sub-data in the semantic comparison space after being mapped should be close during the mapping process. For example, for the example of nodules in the upper left lung above, assuming that the first semantic representation (two-dimensional vector) of the first sub-data of the text mapped to the semantic comparison space (two-dimensional space) is (1.5, 1.8), then the third semantic representation (two-dimensional vector) of the third sub-data of the image mapped to the semantic comparison space should also be (1.5, 1.8) or very close to (1.5, 1.8) (for example, the difference between the two obtained according to the comparison algorithm (such as the cosine similarity described later) is less than or equal to a predetermined threshold), that is, the positions of semantically consistent sub-data mapped to the semantic comparison space should be the same or very close.

[0120] For example, the data of the first modality and the data of the second modality describing the same characteristic of multiple objects (e.g., multiple previous patients) can be used as training data (or data sets), and the first neural network and the second neural network can be trained based on machine learning, so that the sub-data of different modalities (corresponding to the sub-data of different modalities of the same characteristic of the same object) can maintain semantic consistency after being mapped to the semantic comparison space, and thus the relevant parameters of the first neural network and the relevant parameters of the second neural network can be determined. For example, the specific method of training the first neural network and the second neural network through machine learning can refer to a neural network training method and device for determining data similarity provided by at least one embodiment of the present disclosure, which will not be repeated here.

[0121] For example, when the first modality is text and the second modality is an image, the first neural network can be a long short-term memory network (LSTM), or word2vec, or word2vect+LSTM, etc.; the second neural network can be a convolutional neural network (CNN). The embodiments of the present disclosure are not limited to this, as long as the first neural network and the second neural network can respectively process data of different modalities to obtain vectors in the same semantic space, the similarity between the obtained different vectors can be calculated.

[0122] Step S230: obtaining the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation.

[0123] In one example, obtaining the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation, and the fourth semantic representation includes the following steps S231 and S232.

[0124] Step S231: Calculate the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation.

[0125] For example, the similarity sim1(t i ,t j ), the similarity between the third sub-data and the fourth sub-data sim1(g i ,g j ), the similarity sim2(t i ,g j ) and the similarity sim2(g i ,t j) can be calculated using cosine similarity, that is, the similarity of the two sub-data is evaluated by calculating the cosine value of the semantic representation (vector) corresponding to the two sub-data. For example, when the two vectors have the same direction, the cosine similarity value is 1; when the angle between the two vectors is 90°, the cosine similarity value is 0; when the two vectors point in completely opposite directions, the cosine similarity value is -1. Therefore, the cosine similarity (cosine value) of the two vectors is between -1 and 1, and the larger the cosine similarity (cosine value) of the two vectors, the closer the two vectors are. Correspondingly, the higher the similarity of the two sub-data corresponding to the two vectors. For example, when the cosine similarity value is -1, it indicates that the two vectors are negatively correlated. For example, all elements in the vector corresponding to the first sub-data and the vector corresponding to the second sub-data can be made positive. In this case, the cosine similarity of the vector corresponding to the first sub-data and the vector corresponding to the second sub-data is between 0-1.

[0126] For example, the similarity sim1(t i ,t j ), the similarity between the third sub-data and the fourth sub-data sim1(g i ,g j ), the similarity sim2(t i ,g j ) and the similarity sim2(g i ,t j ) can be obtained using the following expression:

[0127]

[0128]

[0129]

[0130]

[0131] Here, the superscript T represents the transpose of a vector, “||” represents the length of a vector, and “·” represents the dot product between two vectors.

[0132] Step S232: Obtain the similarity between the first data and the second data based on the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation.

[0133] For example, the similarity between the first data and the second data can be equal to the sum of the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation.

[0134] For example, the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation are respectively equal to the similarity between the first sub-data and the fourth sub-data, the similarity between the second sub-data and the third sub-data, the similarity between the first sub-data and the second sub-data, and the similarity between the third sub-data and the fourth sub-data.

[0135] For example, the first data p i and the second data p j The similarity between i ,p j ) is obtained using the following expression:

[0136] f(p i ,p j )=f((t i ,g i ),(t j ,g j ))=sim1(t i ,t j )+sim2(t i ,g j )+sim2(g i ,t j )+sim1(g i ,g j ).

[0137] However, the embodiments of the present disclosure are not limited thereto. For example, based on practical experience, suitable weights (between 0 and 1) can be set for the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation, respectively, so as to further improve the first data p obtained by calculation. i and the second data p j The accuracy of the similarity between them.

[0138] For example, step S230 is not limited to being implemented by executing step S231 and step S232. In some examples, step S230 may also be implemented by executing the following steps S233 and S234.

[0139] Step S233: Calculate the similarity between the first semantic representation and the fourth semantic representation and the similarity between the second semantic representation and the third semantic representation.

[0140] Step S234: obtaining the similarity between the first data and the second data based on the similarity between the first semantic representation and the fourth semantic representation and the similarity between the second semantic representation and the third semantic representation. In this case, the similarity between the first data and the second data may be equal to the sum of the similarity between the first semantic representation and the fourth semantic representation and the similarity between the second semantic representation and the third semantic representation, that is, the similarity between the first data and the second data is obtained only by calculating the sub-data of different modalities.

[0141] It should be noted that the specific implementation method of step S230 can be set according to actual application requirements, and the embodiments of the present disclosure do not specifically limit this. For example, step S230 includes: calculating at least one of the similarity between the first semantic representation and the fourth semantic representation and the similarity between the second semantic representation and the third semantic representation. For example, step S230 also includes: calculating at least one of the similarity between the first semantic representation and the second semantic representation and the similarity between the third semantic representation and the fourth semantic representation.

[0142] Figure 3B yes Figure 2 Another example of a method for determining data similarity is shown in FIG. Figure 3B As shown, the first data of the first object includes the first sub-data of the first modality, and the second data of the second object includes the second sub-data of the second modality. For example, the first modality is text and the second modality is image, or the first modality is image and the second modality is text.

[0143] like Figure 3B As shown, the first sub-data of the first modality can be mapped to the semantic comparison space to obtain a first semantic representation through a first neural network, and the second sub-data of the second modality can be mapped to the semantic comparison space to obtain a second semantic representation through a second neural network.

[0144] For example, when the first modality is text and the second modality is an image, the first neural network is a long short-term memory network, or word2vec, or word2vect+LSTM, etc.; the second neural network is a convolutional neural network. Similarly, the embodiments of the present disclosure are not limited to this, as long as the first neural network and the second neural network can respectively process data of different modalities to obtain vectors in the same semantic space, the similarity between the obtained different vectors can be calculated.

[0145] like Figure 3BAs shown, after obtaining the first semantic representation and the second semantic representation, the similarity between the first semantic representation and the second semantic representation is calculated, and the similarity between the first semantic representation and the second semantic representation is used as the similarity between the first data of the first object and the second data of the second object.

[0146] For example, according to actual application requirements, the first data of the first object may further include third sub-data of the second modality, and the second data of the second object may further include fourth sub-data of the first modality.

[0147] It should be noted that at least one embodiment of the present disclosure provides a method for determining data similarity, wherein the first data and the second data are not limited to Figure 3A and Figure 3B The example shown. For example, the first data may also include only the first sub-data of the first modality, and the second data may also include only the second sub-data of the second modality. For another example, the first data may also include only the first sub-data of the second modality, and the second data may also include only the second sub-data of the first modality. For another example, the first data includes the first sub-data of the first modality and the third sub-data of the second modality, and the second data includes only the second sub-data of the first modality or the second modality. For another example, the first data includes only the first sub-data of the first modality or the second modality, and the second data includes the second sub-data of the first modality and the fourth sub-data of the second modality. In these examples, the similarity calculation method between the first data and the second data can be adjusted accordingly according to the application requirements, which will not be elaborated here.

[0148] It should be noted that at least one embodiment of the present disclosure provides a method for determining data similarity, in which either of the first data and the second data is not limited to sub-data of only one or two modalities. According to actual application requirements, at least one of the first data and the second data may also include sub-data of three modalities or sub-data of other applicable numbers of modalities; for example, at least one of the first data and the second data may include sub-data of text, sub-data of image, and sub-data of speech; in this case, the semantic comparison space makes it possible to calculate the similarity of the semantic representation obtained by mapping the sub-data of text to the semantic comparison space, the semantic representation obtained by mapping the sub-data of image to the semantic comparison space, and the semantic representation obtained by mapping the sub-data of speech to the semantic comparison space.

[0149] At least one embodiment of the present disclosure also provides a device for determining data similarity.

[0150] Figure 4 FIG. 1 is an exemplary block diagram of a data similarity determination device 110 provided by at least one embodiment of the present disclosure. Figure 4As shown, the data similarity determination device 110 includes an acquisition module 111 , a mapping module 112 and a similarity calculation module 113 .

[0151] The acquisition module 111 is configured to acquire first data of a first object and second data of a second object; the first data includes first sub-data of a first modality or a second modality, and the second data includes second sub-data of the first modality or the second modality.

[0152] For example, the first object may be a given patient, and the second object may be a reference patient whose similarity with the given patient needs to be determined. For example, the first modality is different from the second modality. For example, the first modality is text, and the second modality is an image.

[0153] In one example, the first data also includes third sub-data for the second modality, and the second data also includes fourth sub-data for the second modality; the first object includes a first characteristic, the first sub-data includes a first sub-semantic describing the first characteristic, and the third sub-data includes a third sub-semantic describing the first characteristic; the second object includes a second characteristic, the second sub-data includes a second sub-semantic describing the second characteristic, and the fourth sub-data includes a fourth sub-semantic describing the second characteristic.

[0154] The mapping module 112 is configured to map the first sub-data into a first semantic representation in a semantic comparison space, and to map the second sub-data into a second semantic representation in a semantic comparison space; the semantic comparison space at least makes it possible to calculate the similarity of the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space.

[0155] For example, the mapping module 112 includes a first sub-mapping module and a second sub-mapping module; the first sub-mapping module is configured to map the sub-data of the first modality to the semantic comparison space, and the second sub-mapping module is configured to map the sub-data of the second modality to the semantic comparison space. For example, the first sub-mapping module can be implemented as a first neural network, and the second sub-mapping module can be implemented as a second neural network. For example, when the first modality is text and the second modality is an image, the first neural network can be a long short-term memory network, or word2vec, or word2vect+LSTM, etc., and the second neural network can be a convolutional neural network. Similarly, the embodiments of the present disclosure are not limited to this, as long as the first neural network and the second neural network can respectively process the data of different modalities to obtain vectors in the same semantic space, thereby calculating the similarity between the obtained different vectors.

[0156] For example, in the case where the first data also includes the third sub-data of the second modality, and the second data also includes the fourth sub-data of the second modality, the mapping module 112 is further configured to map the third sub-data to the third semantic representation in the semantic comparison space, and to map the fourth sub-data to the fourth semantic representation in the semantic comparison space. For example, the first sub-mapping module is configured to map the first sub-data to the first semantic representation of the semantic comparison space and to map the second sub-data to the second semantic representation of the semantic comparison space; the second sub-mapping module is configured to map the third sub-data to the third semantic representation of the semantic comparison space and to map the fourth sub-data to the fourth semantic representation of the semantic comparison space.

[0157] The similarity calculation module 113 is configured to calculate the similarity between the first data and the second data based on at least the first semantic representation and the second semantic representation.

[0158] For example, when the first data also includes third sub-data of the second modality and the second data also includes fourth sub-data of the second modality, the similarity calculation module 113 is configured to obtain the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation.

[0159] For example, the similarity calculation module 113 is configured to calculate the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation. For example, the similarity between the first data and the second data is equal to the sum of the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation.

[0160] For example, a method for calculating the similarity between the first data and the second data may refer to a method for determining data similarity provided in at least one embodiment of the present disclosure, which will not be described in detail herein.

[0161] For example, the acquisition module 111, the mapping module 112, the first sub-mapping module, the second sub-mapping module and the similarity calculation module 113 can be implemented by software, firmware, hardware and any combination thereof, for example, the hardware includes a Field Programmable Gate Array (FPGA) and the like.

[0162] At least one embodiment of the present disclosure also provides another device for determining data similarity.

[0163] Figure 5is an exemplary block diagram of another data similarity determination device provided by at least one embodiment of the present disclosure. Figure 5 As shown, the data similarity determination device includes a processor and a memory. The memory stores computer program instructions suitable for execution by the processor, and when the computer program instructions are executed by the processor, the processor executes any data similarity determination method provided by at least one embodiment of the present disclosure.

[0164] For example, the processor is, for example, a central processing unit (CPU), a graphics processor GPU, a tensor processing unit (TPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. For example, the processor can be implemented as a general-purpose processor, and can also be a single-chip microcomputer, a microprocessor, a digital signal processor, a dedicated image processing chip, or a field programmable logic array, etc. For example, the memory may include at least one of a volatile memory and a non-volatile memory, for example, the memory may include a read-only memory (ROM), a hard disk, a flash memory, etc. Accordingly, the memory may be implemented as one or more computer program products, and the computer program product may include various forms of computer-readable storage media, and one or more computer program instructions may be stored on the computer-readable storage medium. The processor may run the program instructions to execute any data similarity determination method provided by at least one embodiment of the present disclosure. The memory may also store various other applications and various data, for example, various data used and / or generated by the application, etc.

[0165] At least one embodiment of the present disclosure provides a data similarity determination device ( Figure 4 and Figure 5 The data similarity determination device shown) can improve the accuracy of the data similarity calculation results.

[0166] At least one embodiment of the present disclosure also provides a non-transitory storage medium. Figure 6 is an exemplary block diagram of a non-transitory storage medium provided by at least one embodiment of the present disclosure. Figure 6 As shown, the non-transitory storage medium includes computer program instructions stored thereon, and when the computer program instructions are executed by the processor, the computer executes any data similarity determination method provided by at least one embodiment of the present disclosure. For example, the non-transitory storage medium may include a magnetic storage medium, an optical storage medium, a semiconductor storage medium, etc.; for example, the non-transitory storage medium may include a read-only memory (ROM), a hard disk, a flash memory, etc. For example, Figure 6 The non-transitory storage medium shown may be used in data similarity determination and may improve the accuracy of data similarity calculation results.

[0167] At least one embodiment of the present disclosure also provides a method for training a neural network for determining data similarity.

[0168] Figure 7 is a flowchart of a neural network training method for determining data similarity provided by at least one embodiment of the present disclosure. For example, Figure 7 The illustrated training method of a neural network for data similarity determination may be used to train a first neural network and a second neural network.

[0169] like Figure 7 As shown, the training method of the neural network for determining data similarity includes the following steps S310 to S340.

[0170] Step S310: Acquire first training data of a training object, wherein the first training data includes first training sub-data of a first modality and second training sub-data of a second modality, the training object includes training characteristics, the first training sub-data includes first training sub-semantics describing the training characteristics, and the second training sub-data includes second training sub-semantics describing the training characteristics.

[0171] Step S320: Map the first training sub-data into a first training semantic representation in a semantic comparison space through a first neural network, wherein the semantic comparison space enables the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space to be calculable in similarity.

[0172] Step S330: Mapping the second training sub-data into a second training semantic representation in the semantic comparison space through a second neural network.

[0173] Step S340: Calculate the similarity between the first training semantic representation and the second training semantic representation, and adjust the parameters of at least one of the first neural network and the second neural network according to the similarity between the first training semantic representation and the second training semantic representation.

[0174] For example, the first modality is different from the second modality. For example, the first modality is text, and the second modality is an image. For another example, the first modality is an image, and the second modality is text. For example, when the first modality is text and the second modality is an image, the first neural network can be a long short-term memory network, or word2vec, or word2vect+LSTM, etc., and the second neural network can be a convolutional neural network. Similarly, the embodiments of the present disclosure are not limited to this, as long as the first neural network and the second neural network can respectively process data of different modalities to obtain vectors in the same semantic space, the similarity between the obtained different vectors can be calculated.

[0175] For example, the first training semantic representation and the second training semantic representation are both expressed as vectors, that is, the first training semantic representation and the second training semantic representation are both vectors in the semantic comparison space. For example, the dimension of the vector corresponding to the first training semantic representation is the same as the dimension of the vector corresponding to the second training semantic representation. For example, the training characteristic refers to the characteristic to be trained for the first neural network and the second neural network, and the trained first neural network and the second neural network can respectively map the sub-data of the first object corresponding to the training characteristic to the semantic comparison space, and make the dimensions of the vectors corresponding to the obtained semantic representations the same and the values ​​of the vectors corresponding to the semantic representations in each dimension are similar, so that calculations and comparisons can be performed. For example, when training the first neural network and the second neural network, training can be performed for one or more training characteristics.

[0176] For example, cosine similarity may be used to calculate the similarity between the first training semantic representation and the second training semantic representation. For example, the method for calculating cosine similarity may refer to the method for determining data similarity provided in at least one embodiment of the present disclosure, which will not be described in detail here.

[0177] For example, adjusting the parameters of at least one of the first neural network and the second neural network according to the similarity between the first training semantic representation and the second training semantic representation includes: calculating a loss function based on the similarity between the first training semantic representation and the second training semantic representation; and minimizing the loss function by adjusting the parameters of at least one of the first neural network and the second neural network.

[0178] For example, the first training data of M (for example, M is a positive integer greater than 1) training objects (for example, M previous patients) can be used to train the first neural network and the second neural network to determine relevant parameters of the first neural network and relevant parameters of the second neural network. As a result, the first neural network and the second neural network trained using the first training data of the M training objects and the training method of the neural network for determining data similarity can maintain semantic consistency for sub-data of different modalities of the same object (the same first object) after being mapped to the semantic comparison space.

[0179] For example, in step S310, first training data of M training objects may be obtained; in step S320, the first training sub-data of the M training objects may be respectively converted into M first training semantic representations; in step S330, the second training sub-data of the M training objects may be respectively converted into M second training semantic representations; in step S340, the similarities of the first training semantic representations and the second training semantic representations of the M training objects may be respectively calculated, and a loss function (or error function) may be respectively calculated based on the similarities of the first training semantic representations and the second training semantic representations of the M training objects, and then the loss function may be minimized by adjusting the parameters of at least one of the first neural network and the second neural network (for example, adjusting the parameters of the first neural network and the parameters of the second neural network).

[0180] For example, the loss function l can be obtained using the following expression:

[0181]

[0182] Here, trt i is the first training sub-data of the i-th training object, trg i is the second training sub-data of the i-th training object, sim2(trt i ,trg i ) is the similarity between the first training sub-data of the i-th training object and the second training sub-data of the i-th training object, NN1 is the mapping of the corresponding training sub-data using the first neural network, NN2 is the mapping of the corresponding training sub-data using the second neural network, NN1(trt i ) is the first training semantic representation of the i-th training object, NN2(trg i ) is the second training semantic representation of the i-th training object.

[0183] For example, minimizing the loss function by adjusting the parameters of at least one of the first neural network and the second neural network includes: adjusting the parameters of at least one of the first neural network and the second neural network based on a gradient descent method (e.g., a stochastic gradient descent method) or other applicable methods (e.g., a genetic algorithm, a back propagation algorithm). For example, adjusting the parameters of at least one of the first neural network and the second neural network includes adjusting the parameters of at least one of the first neural network and the second neural network multiple times. For example, adjusting the parameters of at least one of the first neural network and the second neural network multiple times can gradually reduce the loss function. For example, when the loss function is less than a loss function threshold (e.g., 0.01) or the number of parameter adjustments is greater than an adjustment number threshold (e.g., 10,000 times), the loss function is considered to be minimized.

[0184] For example, the gradient descent method is used to find the local minimum of a function. For example, the adjustment of the parameters of at least one of the first neural network and the second neural network in order to minimize the loss function (error function) can be regarded as a movement aimed at finding the minimum point on the error plane.

[0185] For example, in the gradient descent algorithm, the learning rate determines the degree to which the weights are modified during the learning process; in the early stages, a higher rate can be set to quickly explore the error plane and quickly approach the optimal solution (the minimum of the plane), and then, in the later stages, the rate is reduced to get as close to the optimal solution as possible. A higher learning rate means faster learning, but there is also a greater risk that the neural network will converge outside the optimal solution. Moment can be used as a damping parameter to reduce oscillations and help achieve convergence; generally speaking, the smaller the moment, the better the network adapts to the environment. Moment makes the weights tend to keep changing in the same direction (increasing or decreasing), and moment limits oscillations caused by irregularities in the learning examples.

[0186] For example, the learning rate of the neural network controls the movement speed, and the moment of the neural network controls the change of direction on the error plane; by moving in the same direction on the error plane for multiple times, it is possible to avoid falling into the local minimum of the loss function (error function), but to reach the global minimum of the loss function (error function) through the local minimum. For example, in the early stages of training a neural network, it is possible to quickly determine in all directions, and then reduce the movement speed at the end, and reduce the frequency of direction changes. For example, the specific method of adjusting the parameters of a neural network based on the gradient descent method (e.g., the stochastic gradient descent method) can be referred to in the relevant technology, which will not be repeated here.

[0187] For example, by adjusting the parameters of at least one of the first neural network and the second neural network so as to minimize the loss function (for example, making the loss function equal to zero or close to zero), the similarity between the first training semantic representation and the second training semantic representation of the training object can be maximized, thereby ensuring that the similarity of sub-data of different modalities of the same training object is maximized, that is, the first neural network and the second neural network trained using the first training data of the M training objects and the training method of the neural network for determining data similarity can maintain semantic consistency for sub-data of different modalities (for example, sub-data of different modalities corresponding to the same first object) after being mapped to the semantic comparison space in the execution of the data similarity determination method.

[0188] For example, according to actual application requirements, the training method of the neural network for determining data similarity also includes the following step S350.

[0189] Step S350: performing data preprocessing on the original data before acquiring the first training data of the training object.

[0190] For example, the original data includes data of N (N is a positive integer greater than or equal to M) objects to be selected, and data preprocessing of the original data includes: selecting objects that meet the following requirements among the N objects to be selected as M training objects: that is, the first training data simultaneously includes the first training sub-data of the first modality and the second training sub-data of the second modality.

[0191] For example, the N candidates may be sorted, and the candidates whose first training data includes both the first training sub-data of the first modality and the second training sub-data of the second modality are placed at the front of the queue. In this case, when acquiring the first training data of the training object, the first M candidates in the original data may be directly selected as the M training objects, thereby simplifying step S310.

[0192] For example, the data similarity determination method can be performed in the order of step S310, step S320, step S330, and step S340, but the embodiments of the present disclosure are not limited thereto. For another example, the data similarity determination method can be performed in the order of step S310, step S320+step S330 (that is, step S320 and step S330 are performed simultaneously), and step S340.

[0193] At least one embodiment of the present disclosure also provides a neural network training device for determining data similarity.

[0194] Figure 8 is an exemplary block diagram of a neural network training device for determining data similarity provided by at least one embodiment of the present disclosure. For example, Figure 8 The illustrated neural network training apparatus for determining data similarity may be used to train a first neural network and a second neural network.

[0195] like Figure 8 As shown, the training device for a neural network for determining data similarity includes a processor and a memory. The memory stores computer program instructions suitable for execution by the processor, and when the computer program instructions are executed by the processor, the processor executes any training method provided by at least one embodiment of the present disclosure. For example, Figure 8 The specific implementation of the processor and memory shown can be found in Figure 5 The processor and memory shown are not described in detail here.

[0196] At least one embodiment of the present disclosure further provides another non-transitory storage medium, which includes computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the computer executes any training method provided by at least one embodiment of the present disclosure. For example, the non-transitory storage medium may include a magnetic storage medium, an optical storage medium, a semiconductor storage medium, etc.; for example, the non-transitory storage medium may include a read-only memory (ROM), a hard disk, a flash memory, etc. For example, another non-transitory storage medium provided by at least one embodiment of the present disclosure can be used to train a first neural network and a second neural network.

[0197] At least one embodiment of the present disclosure also provides a similar object searching method.

[0198] Fig. 9 FIG. 1 is a flowchart of a similar object search method provided by at least one embodiment of the present disclosure. Fig. 9 As shown, the similar object search method includes the following steps S410 to S440.

[0199] Step S410: Acquire first data of a first object.

[0200] Step S420: Acquire second data of multiple second objects.

[0201] Step S430: using any data similarity determination method provided by at least one embodiment of the present disclosure to calculate the similarity between the first object and each second object to obtain multiple object similarities.

[0202] Step S440: outputting information of the second object corresponding to the object similarity with the largest value among the multiple object similarities.

[0203] For example, the first data of the first object can be obtained through the human-computer interaction interface. For example, the first user is a given patient, that is, a patient currently seeking medical treatment. For example, the first data includes first sub-data of the first modality or the second modality. For example, the first data also includes third sub-data of the second modality or the first modality. For example, the first modality is text, and the second modality is an image.

[0204] For example, the second data of multiple second objects can be obtained from the object database. For example, the object database stores data of multiple objects. For example, the multiple second objects are multiple previous patients. For example, the data of each of the multiple objects includes at least one of sub-data of the first modality and sub-data of the second modality. For example, the information of the second object (i.e., similar object) corresponding to the object similarity with the largest value among the multiple object similarities can be output through the human-computer interaction interface (e.g., the information display area of ​​the human-computer interaction interface).

[0205] For example, the specific content of the information of similar objects displayed in the information display area can be set based on the actual needs of the user. For example, the information of similar objects displayed in the information display area can include summary information or important information; for example, the information display area can also display relevant information according to user needs. For example, the above summary information includes object similarity, important information that the user cares about (for example, text information and imaging information on whether there are nodules in the lungs, as well as diagnosis and treatment plans), etc. For example, relevant information includes information related to the above important information (for example, treatment costs, length of hospital stay, etc.).

[0206] For example, in step S440, outputting the information of the second object corresponding to the object similarity with the largest value among multiple object similarities includes: outputting the information of the second object corresponding to the first number of object similarities with the largest value among multiple object similarities. For example, the first number (K) is a positive integer greater than 1. For example, the first number (K) can be set based on user needs, for example, K=5 or 10. For example, by outputting the information of the second object corresponding to the first number of object similarities with the largest value among multiple object similarities, the user can simultaneously refer to the information of the first number of second objects, and artificially confirm the object that is closest to the first object among the first number of second objects, thereby improving the applicability of at least one embodiment of the present disclosure. A similar object search method is also provided.

[0207] For example, the similar object search method can be performed in the order of step S410, step S420, step S430, and step S440, but the embodiments of the present disclosure are not limited thereto. For another example, the similar object search method can be performed in the order of step S310+step S320 (that is, step S410 and step S410 are performed simultaneously), step S430, and step S440.

[0208] For example, according to actual application requirements, the similar object search method further includes the following step S450.

[0209] Step S450: after obtaining the multiple object similarities, before outputting the information of the second object corresponding to the object similarity with the largest value among the multiple object similarities, sorting the multiple second objects based on the multiple object similarities.

[0210] For example, sorting multiple second objects based on multiple object similarities includes sorting the multiple second objects using a sequential sorting method based on the multiple object similarities; in this case, when outputting the information of the second object corresponding to the object similarity with the largest value among the multiple object similarities, the information of the second object at the first position of the sequence (or the first K second objects in the sequence) can be directly output, thereby simplifying step S440.

[0211] For example, by using any data similarity determination method provided by at least one embodiment of the present disclosure to calculate the similarity between a first object and each second object, the accuracy of the data similarity calculation result can be improved, thereby improving the search performance of the similar object search method provided by at least one embodiment of the present disclosure (for example, more similar objects can be found).

[0212] At least one embodiment of the present disclosure further provides a similar object search device, which includes: any data similarity determination device provided by at least one embodiment of the present disclosure.

[0213] Fig.10 is an exemplary block diagram of a similar object search device provided by at least one embodiment of the present disclosure. Fig.10 As shown, the similar object searching device includes any data similarity determining device 110 provided by at least one embodiment of the present disclosure.

[0214] Combine the following Fig.11 and Fig.12 right Fig.10 The similar object searching device shown is used for exemplary description.

[0215] Fig.11 yes Fig.10 An example of a similar object search device is shown. Fig.11 As shown, the similar object search device includes any data similarity determination device 110 provided by at least one embodiment of the present disclosure, a human-computer interaction interface 211, and an object database 212. For example, Fig.11 The similar object searching device shown may be a dedicated device dedicated to similar object searching.

[0216] The human-computer interaction interface 211 is used to realize the interaction between the operator and the computer. For example, it can be realized as a text interaction interface, an image interaction interface, etc., and can be realized with the help of various appropriate types of input and output devices, such as a display, a touch panel, a mouse, a keyboard, a touch ball, etc.

[0217] For example, the human-computer interaction interface 211 is configured to receive first data of a first object and provide the first data to the data similarity determination device 110. The object database 212 stores data of multiple objects and provides data of at least part of the objects to the data similarity determination device 110 as second data of multiple second objects. The data similarity determination device 110 is configured to provide the similarity between the first object and each second object as multiple object similarities; the human-computer interaction interface 211 is also configured to output information of the second object corresponding to the object similarity with the largest value among the multiple object similarities.

[0218] For example, the human-computer interaction interface 211 includes an information input area, and the user inputs the first data of the first object through the information input area, so that the human-computer interaction interface 211 can receive the first data of the first object. For example, the first user is a given patient, that is, a patient currently seeking medical treatment. For example, the first data includes first sub-data that is a first modality or a second modality. For example, the first data also includes third sub-data that is a second modality or a first modality. For example, the first modality is text and the second modality is an image.

[0219] For example, the object database 212 stores data of a plurality of objects, which are a plurality of past patients. For example, the data of each of the plurality of objects includes at least one of sub-data of a first modality and sub-data of a second modality.

[0220] For example, the following uses the example of multiple objects being multiple patients to illustrate the object database (patient database). For example, (t, g) can be used to represent the patient's information, where t represents the patient's medical record data and g represents the patient's imaging data. t or g can be empty, indicating that the patient lacks this part of information. For example, a common flu patient may only have a medical record but no imaging data, in which case g is empty. Assuming that the patient database contains N patient information, the patient database can be recorded as S = {(t1, g1), (t2, g2), ..., (tN, gN)}.

[0221] For example, the object database 212 is a device with a storage function. The object database 212 is mainly used for data of multiple objects. The object database 212 can be local or remote. The object database 212 can include various memories, such as random access memory (Random Access Memory (RAM)), read only memory (Read Only Memory (ROM)), etc.

[0222] For example, after or at the same time as the data similarity determination device 110 receives the first data of the first object, the data similarity determination device 110 obtains data of at least part of the object from the object database 212 as the second data of the plurality of second objects. Fig.11 The data similarity determination device 110 shown provides the similarity between the first object and each second object as the similarity of multiple objects. For a specific implementation manner, reference may be made to the data similarity determination device 110 or data similarity determination method provided in at least one embodiment of the present disclosure, which will not be repeated here.

[0223] For example, the specific manner in which the data similarity determination device 110 obtains data of at least part of the objects from the object database 212 can be set according to actual application requirements. For example, the data similarity determination device 110 can obtain data of a predetermined number of objects from the object database 212 at one time; for another example, the data similarity determination device 110 can also obtain data of one object from the object database 212 each time until the data of the predetermined number of objects is obtained or an object that meets the requirements is found.

[0224] For example, the human-computer interaction interface 211 also includes an information display area; the human-computer interaction interface 211 is also configured to output information of a second object (i.e., a similar object) corresponding to the object similarity with the largest value among multiple object similarities in the display area as a similar object for user reference. For example, outputting information of a second object corresponding to an object similarity with the largest value among multiple object similarities includes: outputting information of a second object corresponding to a first number of object similarities with the largest value among multiple object similarities. For example, the specific content of the information of similar objects displayed in the information display area can be found in at least one embodiment of the present disclosure, which also provides a similar object search method, which is not described in detail here.

[0225] For example, the specific manner in which the human-computer interaction interface 211 obtains and outputs the information of the second object corresponding to the object similarity with the largest value among multiple object similarities can be set according to actual application requirements, and the embodiments of the present disclosure do not specifically limit this. For example, the human-computer interaction interface 211 can receive all the information of the second object corresponding to the object similarity with the largest value among multiple object similarities from the data similarity determination device 110, and only display the information required by the user according to the user settings. For another example, the human-computer interaction interface 211 can also receive the number and object similarity of the second object corresponding to the object similarity with the largest value among multiple object similarities from the data similarity determination device 110, and then obtain the information required by the user from the object database 212 based on the number of the second object, and then display the information required by the user and the object similarity in the display area; for example, when the user needs to view further information about the object, the human-computer interaction interface 211 again obtains the new information required by the user from the object database based on the number of the second object; in this case, the Fig.11 The overall traffic of the similar object search device is shown.

[0226] For example, since the first data of the first object and the second data of the second object are both mapped to the semantic comparison space in which the semantic representations corresponding to the sub-data of different modalities can be calculated, Fig.11 The similar object searching device can utilize the information complementarity of data of different modalities and / or the characteristics of verifying and checking each other for data of different modalities, thereby Fig.11The similar object searching device shown can improve the performance of searching for similar objects.

[0227] Fig.12 yes Fig.10 Another example of a similar object search device is shown. Fig.12 As shown, the similar object search device includes an information sending and receiving device 221, an object database 222, and any data similarity determination device 110 provided by at least one embodiment of the present disclosure. For example, Fig.12 The functions of the similar object search device shown can be implemented by a server, which can be a single server or a server group, and each server in the server group is connected via a wired network or a wireless network. The wired network can communicate by means of twisted pair, coaxial cable or optical fiber transmission, for example, and the wireless network can communicate by means of 3G / 4G / 5G mobile communication network, Bluetooth, Zigbee or WiFi, for example. The present disclosure does not limit the type and function of the network. The server group can be centralized, such as a data center, or distributed. The server can be local or remote. For example, the server can be a general-purpose server or a dedicated server, and can be a virtual server or a cloud server, etc.

[0228] For example, the information transmitter and receiver 221 may include a modem, a network adapter, a Bluetooth transmitter and receiver unit, or an infrared transmitter and receiver unit, etc. For example, the information transmitter and receiver 221 may also perform operations such as encoding and decoding on the information to be transmitted or received.

[0229] For example, Fig.12 The information transmitter and receiver 221 shown is configured to receive the first data of the first object from the client, and provide the first data to the data similarity determination device 110. For example, the information transmitter and receiver 221 can realize the sending and receiving of information through, for example, a network or other technologies. For example, the network can be a single network, or a combination of at least two different networks. For example, the network can include but is not limited to a local area network, a wide area network, a public network, a private network, the Internet, a mobile communication network, etc., or a combination of several thereof, and the other technologies can include, for example, Bluetooth communication, infrared communication, etc.

[0230] For example, the object database 222 stores data of multiple objects, where the multiple objects are multiple past patients. For example, the specific implementation of the object database 222 is shown in Fig.11 The object database 212 shown is not described in detail here.

[0231] For example, in Fig.12After the data similarity determination device 110 receives the first data of the first object, the data similarity determination device 110 obtains data of at least part of the object from the object database 222 as the second data of the plurality of second objects. Fig.12 The data similarity determination device 110 shown is configured to provide the similarity between the first object and each second object as a plurality of object similarities. Fig.12 The data similarity determination device 110 shown provides the similarity between the first object and each second object as the similarity of multiple objects. For a specific implementation manner, reference may be made to the data similarity determination device 110 or data similarity determination method provided in at least one embodiment of the present disclosure, which will not be repeated here.

[0232] For example, the information sending and receiving device 221 is further configured to receive information about a second object corresponding to the object similarity with the largest value among multiple object similarities, and provide the information about the second object corresponding to the object similarity with the largest value among multiple object similarities to the client. For example, providing information about the second object corresponding to the object similarity with the largest value among multiple object similarities includes: providing information about the second object corresponding to the first number of object similarities with the largest value among multiple object similarities.

[0233] For example, the specific manner in which the information sending and receiving receiver 221 receives and provides the information of the second object corresponding to the object similarity with the largest value among multiple object similarities can be set according to actual application requirements, and the embodiments of the present disclosure do not specifically limit this. For example, the information sending and receiving receiver 221 can receive all the information of the second object corresponding to the object similarity with the largest value among multiple object similarities from the data similarity determination device 110, and provide all the information of the second object corresponding to the object similarity with the largest value among the above multiple object similarities to the client, and the client only displays the information required by the user according to the user settings. For another example, the information sending and receiving receiver 221 can also receive the number and object similarity of the second object corresponding to the object similarity with the largest value among multiple object similarities from the data similarity determination device 110, and then obtain the information currently required by the user from the object database 222 based on the number of the second object, and provide the information currently required by the user and the object similarity to the client; for example, when the user needs to view further information about the object, the information sending and receiving receiver 221 again obtains the new information required by the user from the object database 222 based on the number of the second object; in this case, the time required for the information to be sent to the client can be reduced. Fig.12 The overall traffic of the similar object search device is shown.

[0234] At least one embodiment of the present disclosure also provides a similar object search device, which includes: a human-computer interaction interface and an information transmitter and receiver. The human-computer interaction interface is configured to receive first data of a first object and provide the first data to a server; the information transmitter and receiver is further configured to receive second data of a first number of second objects with the greatest similarity to the first data of the first object from the server; the similarity between the first object and each second object is determined by any data similarity determination device provided by at least one embodiment of the present disclosure.

[0235] Fig.13 is an exemplary block diagram of another similar object search device provided by at least one embodiment of the present disclosure. Fig.13 As shown, the similar object search device includes a human-computer interaction interface 232 and an information sending and receiving device 231. For example, Fig.13 The similar object search device shown may be a component of other electronic devices (eg, mobile phones, tablet computers, laptop computers, etc.). For another example, Fig.13 The similar object searching device shown may also be a dedicated device dedicated to similar object searching.

[0236] For example, the human-computer interaction interface 232 is configured to receive first data of a first object and provide the first data to the information transmitter and receiver 231. For example, the specific implementation method of the human-computer interaction interface 232 can be found in Fig.11 The human-computer interaction interface shown will not be described in detail here.

[0237] For example, the information transmitter and receiver 231 is configured to provide the first data to the server; the information transmitter and receiver 231 is also configured to receive the second data of the first number of second objects with the greatest similarity to the first data of the first object from the server, and provide the second data of the first number of second objects with the greatest similarity to the first data of the first object to the human-computer interaction interface 232. For example, providing the information of the second object corresponding to the object similarity with the greatest value among multiple object similarities includes: providing the information of the second object corresponding to the first number of object similarities with the greatest value among multiple object similarities.

[0238] For example, the information transmitter and receiver 231 can realize the transmission and reception of information through a network or other technologies, for example, the network can be a single network, or a combination of at least two different networks. For example, the network can include but is not limited to a local area network, a wide area network, a public network, a private network, the Internet, a mobile communication network, etc., or a combination of several of them. For example, the information transmitter and receiver 231 can include a modem, a network adapter, a Bluetooth transmitter and receiver unit, or an infrared transmitter and receiver unit, etc. For example, the information transmitter and receiver 231 can also perform operations such as encoding and decoding on the information transmitted or received.

[0239] For example, the similarity between the first object and each second object is determined by any data similarity determination device provided by at least one embodiment of the present disclosure. Fig.13 The specific calculation method of the similarity between the first object and each second object obtained by the similar object search device shown can refer to the data similarity determination device or data similarity determination method provided by at least one embodiment of the present disclosure, which will not be repeated here.

[0240] For example, Fig.12 The similar object search device shown acts as a server and Fig.13 The similar object search device shown as a client can cooperate with each other to form a Fig.14 In the similar object searching system shown, the client and the server may be located at the same site or at different sites.

[0241] Fig.15 FIG. 1 shows an exemplary scene diagram of a similar object search system provided by at least one embodiment of the present disclosure. Fig.15 As shown, the similar object search system 300 may include a user terminal 310 , a network 320 , a server 330 , and a database 340 .

[0242] For example, the user terminal 310 may be Fig.15 The computer 310-1 and the portable terminal 310-2 are shown in FIG. It is understandable that the user terminal may also be any other type of electronic device capable of receiving, processing and displaying data, which may include but is not limited to desktop computers, laptop computers, tablet computers, smart home devices, wearable devices, vehicle-mounted electronic devices, monitoring devices, medical electronic devices, etc.

[0243] The user terminal provided according to at least one embodiment of the present disclosure can be used to receive the first data of the first object. For example, the user terminal can receive the first data of the first object via a human-computer interaction interface on the user terminal. For another example, the user terminal can also receive the first data of the first object from a server via a network. For another example, the user terminal can also receive the first data of the first object via a network or directly from a data acquisition device such as a medical acquisition device.

[0244] In some embodiments, the data similarity determination method provided by at least one embodiment of the present disclosure can be executed by a processing unit of a user terminal. In this case, the user terminal has a database storing multiple objects (i.e., an object database), or communicates with the object database via a network or other technology to obtain the second data of multiple second objects. In some implementations, the user terminal can execute the data similarity determination method provided by at least one embodiment of the present disclosure using an application built into the user terminal. In other implementations, the user terminal can execute the data similarity determination method provided by at least one embodiment of the present disclosure by calling an application stored externally to the user terminal.

[0245] In other embodiments, the user terminal sends the received first data to the server 330 via the network 320 or other technologies, and the server 330 obtains the second data of multiple second objects from the object database, and executes the data similarity determination method provided by at least one embodiment of the present disclosure to obtain the similarity between the first data of the first object and the second data of the second object; then, the user terminal can receive and display the similarity between the first data of the first object and the second data of the second object provided by the server 330 and the information of the similar objects determined based on the similarity. In some implementations, the server 330 can use the application built into the server to execute the data similarity determination method provided by at least one embodiment of the present disclosure.

[0246] In other implementations, the server 330 may execute the data similarity determination method provided by at least one embodiment of the present disclosure by calling an application program stored outside the server.

[0247] For example, the specific implementation of the network 320, other technologies and the server 330 may refer to the network, other technologies and servers in the above-mentioned embodiments, which will not be described in detail here.

[0248] The database 340 may include an object database provided by at least one embodiment of the present disclosure to store data of multiple objects. For example, the database 330 may also be used to store various data used, generated, and outputted from the user terminal 310 and the server 330 during operation. The database 340 may be interconnected or communicated with the server 330 or a part of the server 330 via the network 320, or directly interconnected or communicated with the server 330, or interconnected or communicated with the server 330 via a combination of the above two methods.

[0249] In some embodiments, the database 340 may be an independent device. In other embodiments, the database 340 may also be integrated in at least one of the user terminal 310 and the server 340. For example, the database 340 may be set on the user terminal 310 or on the server 340. For another example, the database 340 may also be distributed, with a portion of it being set on the user terminal 310 and another portion being set on the server 340.

[0250] The method, device or system according to the embodiments of the present application can also be Fig.16 The architecture of the computing device 400 shown is implemented.

[0251] Fig.16 The architecture of the computing device 400 is shown. Fig.16 As shown, the computing device 400 may include a bus 410, one or at least two CPUs 420, a read-only memory (ROM) 430, a random access memory (RAM) 440, a communication port 450 connected to a network, an input / output component 460, a hard disk 470, etc. The storage device (e.g., ROM 430 or hard disk 470) in the computing device 400 may store at least one corresponding instruction and various related data or files in the data similarity determination method, the training method of the neural network for data similarity determination, and the similar object search method provided by at least one embodiment of the present disclosure. The computing device 400 may also include a human-machine user interface 480.

[0252] certainly, Fig.16 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig.16 One or at least two components of the computing device shown.

[0253] Although the disclosure has been described in detail above with general descriptions and specific implementation methods, it is obvious to those skilled in the art that some modifications or improvements may be made to the embodiments of the disclosure. Therefore, these modifications or improvements made without departing from the spirit of the disclosure are within the scope of protection claimed by the disclosure.

[0254] The above description is merely an exemplary embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure. The protection scope of the present disclosure is determined by the appended claims.

Claims

1. A method for determining data similarity, comprising: Acquire first data of a first object, wherein the first data includes first sub-data of a first modality or a second modality; Mapping the first sub-data to a first semantic representation in a semantic comparison space, wherein the semantic comparison space allows a semantic representation obtained by mapping the data of the first modality to the semantic comparison space and a semantic representation obtained by mapping the data of the second modality to the semantic comparison space to be similar; Acquire second data of a second object, wherein the second data includes second sub-data of the first modality or the second modality; Mapping the second sub-data into a second semantic representation in the semantic comparison space; The first data further includes third sub-data of the second modality; Mapping the third sub-data into a third semantic representation in the semantic comparison space; The second data further includes fourth sub-data of the second modality; Mapping the fourth sub-data into a fourth semantic representation in the semantic comparison space; and calculating the similarity between the first data and the second data based at least on the first semantic representation, the second semantic representation, the third semantic representation, and the fourth semantic representation; Wherein, the first semantic representation, the second semantic representation, the third semantic representation and the fourth semantic representation are all expressed as vectors; The first semantics indicates the dimension of the corresponding vectors, the second semantics indicates the dimension of the corresponding vectors, the third semantics indicates the dimension of the corresponding vectors, and the fourth semantics indicates that the dimensions of the corresponding vectors are equal to each other; and The similarity f(p) between the first data and the second data i ,p j ) is obtained using the following expression: f(p i ,p j )=f((t i ,g i ),(t j ,g j ))=sim1(t i ,t j )+sim2(t i ,g j )+sim2(g i ,t j )+sim1(g i ,g j ), rt i =NN1(t i ),rg i =NN2(g i ),rt j =NN1(t j ),rg j =NN2(g j ), Among them, p i is the first data, p j For the second data, t i is the first sub-data, g i is the third sub-data, t j is the second sub-data, g j is the fourth sub-data, sim1(t i ,t j ) is the similarity between the first sub-data and the second sub-data, sim1(g i ,g j ) is the similarity between the third sub-data and the fourth sub-data, sim2(t i ,g j ) is the similarity between the first sub-data and the fourth sub-data, sim2(g i ,t j ) is the similarity between the second sub-data and the third sub-data, NN1 uses the first neural network to map the corresponding sub-data, and NN2 uses the second neural network to map the corresponding sub-data. rt i For the first semantic representation, rg i is the third semantic representation, rt j For the second semantic representation, rg j is the fourth semantic representation; wherein the first sub-data is mapped to the first semantic representation and the second sub-data is mapped to the second semantic representation using a first neural network; Using a second neural network, mapping the third sub-data into the third semantic representation and mapping the fourth sub-data into the fourth semantic representation; and Obtaining the similarity between the first data and the second data based on the first semantic representation, the second semantic representation, the third semantic representation, and the fourth semantic representation: calculating the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation; and The similarity between the first data and the second data is obtained based on the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation.

2. The method for determining data similarity according to claim 1, wherein: The first modality is text, and the second modality is image.

3. The method for determining data similarity according to claim 1, wherein: The first object includes a first characteristic, the first sub-data includes a first sub-semantic describing the first characteristic, and the third sub-data includes a third sub-semantic describing the first characteristic; The second object includes a second characteristic, the second sub-data includes second sub-semantics describing the second characteristic, and the fourth sub-data includes fourth sub-semantics describing the second characteristic.

4. The method for determining data similarity according to claim 1, wherein: The similarity between the first data and the second data is equal to the sum of the similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation.

5. The method for determining data similarity according to claim 1, wherein: The similarity between the first semantic representation and the fourth semantic representation, the similarity between the second semantic representation and the third semantic representation, the similarity between the first semantic representation and the second semantic representation, and the similarity between the third semantic representation and the fourth semantic representation are respectively equal to the similarity between the first sub-data and the fourth sub-data, the similarity between the second sub-data and the third sub-data, the similarity between the first sub-data and the second sub-data, and the similarity between the third sub-data and the fourth sub-data.

6. A data similarity determination device, comprising: processor and memory, The memory stores computer program instructions suitable for execution by the processor, and when the computer program instructions are executed by the processor, the processor executes the method for determining data similarity as described in any one of claims 1 to 5.

7. A non-transitory storage medium comprising computer program instructions stored thereon, in, When the computer program instructions are executed by a processor, the computer executes the data similarity determination method according to any one of claims 1 to 5.

8. A method for training a neural network for determining data similarity, comprising: Acquire first training data of a training object, wherein the first training data includes first training sub-data of a first modality and second training sub-data of a second modality, the training object includes training characteristics, the first training sub-data includes first training sub-semantics describing the training characteristics, and the second training sub-data includes second training sub-semantics describing the training characteristics; Mapping the first training sub-data into a first training semantic representation in a semantic comparison space through a first neural network, wherein the semantic comparison space enables the semantic representation obtained by mapping the data of the first modality to the semantic comparison space and the semantic representation obtained by mapping the data of the second modality to the semantic comparison space to be similar to each other; Mapping the second training sub-data into a second training semantic representation in the semantic comparison space through a second neural network; Calculating a similarity between the first training semantic representation and the second training semantic representation, and adjusting a parameter of at least one of the first neural network and the second neural network according to the similarity between the first training semantic representation and the second training semantic representation; The step of adjusting the parameters of at least one of the first neural network and the second neural network according to the similarity between the first training semantic representation and the second training semantic representation comprises: calculating a loss function based on the similarity between the first training semantic representation and the second training semantic representation, and Minimizing the loss function by adjusting parameters of at least one of the first neural network and the second neural network so that the loss function is equal to or close to zero; and The loss function l is obtained using the following expression: Among them, trt i is the first training sub-data of the i-th training object, trg i is the second training sub-data of the i-th training object, sim2(trt i ,trg i ) is the similarity between the first training sub-data of the i-th training object and the second training sub-data of the i-th training object, NN1 uses the first neural network to map the corresponding training sub-data, and NN2 uses the second neural network to map the corresponding training sub-data. NN1(trt i ) is the first training semantic representation of the i-th training object, NN2(trg i ) is the second training semantic representation of the i-th training object, M is the number of training objects, M is a positive integer greater than 1, and N is a positive integer greater than or equal to M; The minimizing the loss function by adjusting the parameters of at least one of the first neural network and the second neural network comprises adjusting the parameters of at least one of the first neural network and the second neural network based on a gradient descent method to maximize the similarity between the first training semantic representation and the second training semantic representation of the training object; The first training sub-data is mapped to the first training semantic representation and the second training sub-data is mapped to the second training semantic representation using the first neural network.

9. A neural network training system for determining data similarity, comprising: processor and memory, The memory stores computer program instructions suitable for execution by the processor, and when the computer program instructions are executed by the processor, the processor executes the training method as claimed in claim 8.

10. A similar object search method, comprising: Acquire first data of a first object; Acquire second data of multiple second objects; Calculate the similarity between the first object and each of the second objects using the data similarity determination method according to any one of claims 1 to 5 to obtain multiple object similarities; Output the information of the second object corresponding to the object similarity with the largest value among the multiple object similarities.

11. A similar object search device, comprising: The data similarity determination device as claimed in claim 6.