Lost and found method based on multi-modal learning and related device

By matching passenger descriptions and found image features using a multimodal learning network, the problem of low efficiency in airport lost and found services has been solved, realizing intelligent lost and found services and reducing the consumption of human resources.

CN118968256APending Publication Date: 2024-11-15SHENYANG CIVIL AVIATION NORTHEAST KAIYA CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411201244.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Airport lost and found services are inefficient, require a lot of manpower, and cause inconvenience to passengers.

Method used

A multimodal learning-based approach is adopted, which trains a multimodal learning network using a deep Boltzmann machine and uses the features of the passenger's descriptive text and the image features of the picked-up item to match and identify the target lost item.

Benefits of technology

It improves the efficiency of lost and found services, reduces the consumption of human resources, and provides intelligent lost and found services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968256A_ABST
    Figure CN118968256A_ABST
Patent Text Reader

Abstract

The invention discloses a lost and found method based on multi-modal learning and a related device, and relates to the technical field of software, and the method comprises the steps: obtaining a first description text of a passenger for a target lost article, and extracting a first text feature of the target lost article from the first description text; inputting the first text feature into a multi-modal learning network to obtain a first image feature output by the multi-modal learning network; wherein the multi-modal learning network is obtained by training a deep Boltzmann machine by taking a second description text for calibrating the lost property and a lost property image as training data in advance; acquiring an object picking image of the picked object, and extracting a second image feature of the picked object from the object picking image; and determining a target lost article from the picked-up articles by matching the first image feature with the second image feature. Intelligent lost and found can be realized, the efficiency is improved, and convenience is brought to passengers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of software technology, and in particular to a lost and found method based on multimodal learning and a related device. Background Art

[0002] With the development of the economy and the progress of the times, more and more passengers choose to travel by plane. As the number of travelers increases, passengers losing their belongings at airports also occurs frequently.

[0003] At present, the method of handling lost and found in airports is basically manual consultation and assistance in searching, which requires a lot of manpower and results in low efficiency, bringing a lot of inconvenience to passengers. Summary of the invention

[0004] In view of the above problems, this application provides a lost and found method and related device based on multimodal learning to achieve the purpose of providing intelligent lost and found services for passengers. The specific scheme is as follows:

[0005] The first aspect of the present application provides a lost and found method based on multimodal learning, and the lost and found method based on multimodal learning includes:

[0006] Acquire a first description text of the target lost property by the passenger, and extract a first text feature of the target lost property from the first description text;

[0007] Inputting the first text feature into a multimodal learning network to obtain a first image feature output by the multimodal learning network; wherein the multimodal learning network is obtained by training a deep Boltzmann machine in advance using the second description text of the calibrated lost object and the lost object image as training data;

[0008] Acquire a picked-up object image, and extract a second image feature of the picked-up object from the picked-up object image;

[0009] The target lost object is determined from the found objects by matching the first image feature and the second image feature.

[0010] In a possible implementation, the process of training a deep Boltzmann machine using the second description text and the lost object image of the calibrated lost object as training data to obtain the multimodal learning network includes:

[0011] Constructing a basic model, wherein the basic model includes a first deep Boltzmann machine and a second deep Boltzmann machine, wherein the first deep Boltzmann machine and the second deep Boltzmann machine establish a joint representation through a joint layer;

[0012] Determine the sample lost objects used for this iterative training from the calibrated lost objects;

[0013] Acquire a second description text of the sample lost object, and extract a second text feature of the sample lost object from the second description text; and acquire a lost object image of the sample lost object, and extract a third image feature of the sample lost object from the lost object image;

[0014] Input the second text feature into the first deep Boltzmann machine, and input the third image feature into the second deep Boltzmann machine, iteratively update the weight parameters of the first deep Boltzmann machine, the second deep Boltzmann machine and the joint layer by calculating the probability distribution, and return to the step of determining the sample lost items for this iterative training from the calibrated lost items, until the preset end condition is met, and then stop the iterative training;

[0015] The trained basic model is used as the multimodal learning network.

[0016] In a possible implementation, the step of acquiring a picked-up image of the picked-up object includes:

[0017] Get the flight date of the passenger;

[0018] Get the first pickup date of the recorded pickup;

[0019] Determine, from the picked-up items, a first target picked-up item whose first picking-up date is later than the flight date;

[0020] Acquire the object image of the first target object.

[0021] In a possible implementation, the determining the target lost object from the found objects by matching the first image feature and the second image feature includes:

[0022] calculating a Euclidean distance between the first image feature and the second image feature;

[0023] Determine a second target object with the smallest Euclidean distance among the objects;

[0024] The picked-up image of the second target picked-up object is pushed to the passenger, so that the passenger can identify the target picked-up object therefrom.

[0025] In a possible implementation, the determining the target lost object from the found objects by matching the first image feature and the second image feature further includes:

[0026] Determining whether the number of the second target objects is greater than a preset number threshold;

[0027] If not, the step of pushing the picked-up image of the second target picked-up object to the passenger is performed so that the passenger can identify the target picked-up object therefrom.

[0028] In a possible implementation, the determining the target lost object from the found objects by matching the first image feature and the second image feature further includes:

[0029] If yes, obtain the loss date of the target lost object and the movement location of the passenger in the airport, and obtain the second pick-up date and pick-up location of the second target found object, wherein the movement location and the pick-up location are both located in three-dimensional space;

[0030] Determine the time difference between the lost date and the second picked-up date, and the shortest distance between the moving position point and the picked-up position point;

[0031] Calculate the matching probability between the second target found object and the target lost object according to the time difference and the shortest distance, wherein the matching probability is negatively correlated with the time difference and the matching probability is negatively correlated with the shortest distance;

[0032] Matching the third target objects with the highest probability from the second target objects, the number of the third target objects being the same as the number threshold;

[0033] The picked-up image of the third target picked-up object is pushed to the passenger, so that the passenger can identify the target picked-up object therefrom.

[0034] A second aspect of the present application provides a lost and found device based on multimodal learning, and the lost and found device based on multimodal learning includes:

[0035] A multimodal learning module, used to train a deep Boltzmann machine using a second description text and an image of the lost object as training data to obtain the multimodal learning network;

[0036] A lost property feature acquisition module, used to acquire a first description text of a target lost property by a passenger, and extract a first text feature of the target lost property from the first description text; input the first text feature into the multimodal learning network to obtain a first image feature output by the multimodal learning network;

[0037] The lost object determination module is used to obtain a found object image of the found object and extract a second image feature of the found object from the found object image; and determine the target lost object from the found object by matching the first image feature with the second image feature.

[0038] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the lost and found method based on multimodal learning according to the first aspect or any implementation of the first aspect.

[0039] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program so that the electronic device can implement the lost and found method based on multimodal learning of the above-mentioned first aspect or any implementation manner of the first aspect.

[0042] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the lost and found method based on multimodal learning according to the first aspect or any implementation method of the first aspect.

[0043] By means of the above technical scheme, the present application provides a lost and found method and related device based on multimodal learning, which obtains the first description text of the target lost object by the passenger, and extracts the first text feature of the target lost object from the first description text; inputs the first text feature into the multimodal learning network to obtain the first image feature output by the multimodal learning network; wherein the multimodal learning network is obtained by pre-training the deep Boltzmann machine with the second description text and the lost object image of the calibrated lost object as training data; obtains the picked-up image of the picked-up object, and extracts the second image feature of the picked-up object from the picked-up image; determines the target lost object from the picked-up object by matching the first image feature and the second image feature. By training the multimodal learning network, the present application can output the corresponding image feature based on the text feature of the description text when the passenger loses the item, and then matches it with the image feature of the picked-up item, so as to determine the passenger's lost item, which can realize intelligent lost and found, improve efficiency and facilitate passengers. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0045] Figure 1 A flowchart of a lost and found method based on multimodal learning provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the structure of a basic model of a multimodal learning network provided in an embodiment of the present application;

[0047] Figure 3 A partial flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application;

[0048] Figure 4 Another partial flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application;

[0049] Figure 5 Another partial flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application;

[0050] Figure 6 Another partial flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of a multimodal learning lost and found device provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0054] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0055] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0056] The embodiment of the present application provides a lost and found method based on multimodal learning. The lost and found method based on multimodal learning in the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0057] See also Figure 1 , Figure 1 The following is a flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application. Figure 1 As shown, a lost and found method based on multimodal learning provided in an embodiment of the present application may include steps S10 to S40, and these steps are described in detail below.

[0058] S10, obtaining a first description text of the target lost property by the passenger, and extracting a first text feature of the target lost property from the first description text.

[0059] In an embodiment of the present application, a passenger can submit a lost and found request through an application, telephone, or on-site, which includes a first description text of the target lost item. The first description text is used to describe the appearance of the target lost item, such as item type, item color, item shape, item surface pattern, item identification (such as a brand logo), etc.

[0060] Based on this, the first description text can be segmented and classified to obtain descriptive word tags for the target lost object, such as the object type is a thermos cup, the object color is red, etc. Then, the descriptive word tags are transformed into vector representations using technologies such as word embedding to obtain the first text features of the target lost object.

[0061] S20, inputting the first text feature into the multimodal learning network to obtain the first image feature output by the multimodal learning network; wherein the multimodal learning network is obtained by pre-training a deep Boltzmann machine using the second description text of the calibrated lost object and the image of the lost object as training data.

[0062] In the embodiment of the present application, the deep Boltzmann machine is used as the basic model, and the second description text of the calibrated lost object (i.e., the calibrated lost object) and the captured image of the lost object are used as training data, and the basic model is iteratively trained to obtain a multimodal learning network. It should be noted that the Deep Boltzmann Machine (DBM) is a neural network model that can learn the probability distribution of high-dimensional data and generate new data points through unsupervised learning.

[0063] In this application, the multimodal learning network obtained by training the deep Boltzmann machine with the calibrated second description text and the lost object image as training data can map the features of the two modalities of the second description text and the lost object image into the same vector space, so as to learn the joint probability distribution of the text features and image features of the same lost object. Based on this, after the multimodal learning network obtains the first text feature, it can predict the output of the corresponding first image feature based on the joint probability distribution.

[0064] S30, acquiring a picked-up object image of the picked-up object, and extracting a second image feature of the picked-up object from the picked-up object image.

[0065] In the embodiment of the present application, for the found items that have been entered at the lost and found office, the corresponding found item image can be photographed, and a certain feature extraction algorithm (such as HOG, LBP, etc.) can be used to extract the second image feature of the found item from the found item image.

[0066] S40, determining the target lost object from the found objects by matching the first image feature and the second image feature.

[0067] In the embodiment of the present application, the similarity between the first image feature and the second image feature can be calculated to determine the matching degree between the found object and the target lost object. Of course, the higher the similarity, the higher the matching degree. Based on this, one or more found objects with the highest matching degree are selected, and the passenger determines his target lost object from them.

[0068] In a possible implementation, the process of training the deep Boltzmann machine using the second description text and the lost object image of the calibrated lost object as training data to obtain the multimodal learning network may include the following steps:

[0069] Construct a basic model, which includes a first deep Boltzmann machine and a second deep Boltzmann machine, and the first deep Boltzmann machine and the second deep Boltzmann machine establish a joint representation through a joint layer;

[0070] Determine the sample lost objects used for this iteration training from the calibrated lost objects;

[0071] Acquire a second description text of the sample lost object, and extract a second text feature of the sample lost object from the second description text; and acquire a lost object image of the sample lost object, and extract a third image feature of the sample lost object from the lost object image;

[0072] Input the second text feature into the first deep Boltzmann machine, and input the third image feature into the second deep Boltzmann machine, iteratively update the weight parameters of the first deep Boltzmann machine, the second deep Boltzmann machine, and the joint layer by calculating the probability distribution, and return to the step of determining the sample lost items for this iterative training from the calibrated lost items, until the preset end condition is met, and then stop the iterative training;

[0073] The trained base model is used as a multimodal learning network.

[0074] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the basic model of the multimodal learning network provided in the embodiment of the present application. Figure 2 As shown, the basic model includes two deep Boltzmann machines, which establish a joint representation through a joint layer H3. Each deep Boltzmann machine consists of a visible layer and two hidden layers. The first deep Boltzmann machine includes a visible layer Vt, a hidden layer Ht1 and a hidden layer Ht2, and the second deep Boltzmann machine includes a visible layer Vm, a hidden layer Hm1 and a hidden layer Hm2. The hidden layer Ht2 and the hidden layer Hm2 serve as inputs of the joint layer H3, and the final joint probability distribution is learned by the joint layer H3. In addition, Wh1 represents the symmetric interaction parameters of the visible layer Vt and the hidden layer Ht1, Wh2 represents the symmetric interaction parameters of the hidden layer Ht1 and the hidden layer Ht2, Wm1 represents the symmetric interaction parameters of the visible layer Vm and the hidden layer Hm1, Wm2 represents the symmetric interaction parameters of the hidden layer Hm1 and the hidden layer Hm2, Wh3 represents the symmetric interaction parameters of the joint layer H3 and the hidden layer Ht2, and Wm3 represents the symmetric interaction parameters of the joint layer H3 and the hidden layer Hm2.

[0075] In the process of training the basic model, at each iteration, the sample lost object used for this iteration training can be determined from the calibrated lost objects, and then the second description text and lost object image of the sample lost object can be obtained. The second description text is manually labeled and the lost object image is pre-photographed. In practical applications, since the devices used to take different lost object images may be different, the size of the lost object images cannot be completely unified. For this purpose, the lost object images of all sample lost objects can be normalized to facilitate calculation. For example, the size of the lost object image can be set to 128*128.

[0076] The second description text is segmented and classified to obtain the descriptive word labels of the sample lost object, and then the descriptive word labels are transformed into vector representation using word embedding and other technologies to obtain the second text features of the sample lost object. Similarly, certain feature extraction algorithms (such as HOG, LBP, etc.) can also be used to extract the corresponding third image features of the lost object image.

[0077] The multimodal learning process is divided into two stages: forward propagation and back propagation. In the forward propagation stage, the second text feature is input into the first deep Boltzmann machine, that is, the second text feature is used as the input data of the visible layer Vt, and the probability distribution of the hidden layer Ht1 and the hidden layer Ht2 can be calculated; in the back propagation stage, the weight parameters (including Wh1 and Wh2) of each layer of neurons can be updated according to the probability distribution of the hidden layer Ht1 and the hidden layer Ht2. Similarly, in the forward propagation stage, the third image feature is input into the second deep Boltzmann machine, that is, the third image feature is used as the input data of the visible layer Vm, and the probability distribution of the hidden layer Hm1 and the hidden layer Hm2 can be calculated; in the back propagation stage, the weight parameters (including Wm1 and Wm2) of each layer of neurons can be updated according to the probability distribution of the hidden layer Hm1 and the hidden layer Hm2.

[0078] Of course, in the forward propagation stage, the joint layer H3 can calculate its joint probability distribution according to the probability distribution of the hidden layer Ht2 and the hidden layer Hm2. Correspondingly, in the backward propagation stage, the joint layer H3 can update the weight parameters of its neurons (including Wh3 and Wm3) according to the joint probability distribution. In this way, the process of iteratively updating the weight parameters of the first deep Boltzmann machine, the second deep Boltzmann machine and the joint layer in this iterative training process is completed.

[0079] If the end condition is not met (for example, the number of iterations is not reached and the convergence condition is not met), the next iteration training will be continued. On the contrary, if the end condition is met (for example, the number of iterations is reached or the convergence condition is not met), the training is terminated and the trained basic model is used as the multimodal learning network.

[0080] In a possible implementation, in order to reduce the usage of computing resources, the items found can also be filtered according to the flight date of the passenger. Figure 3 , Figure 3 A partial flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application. Figure 3 As shown, an embodiment of the present application provides a lost and found method based on multimodal learning, wherein "obtaining a found object image" in step S30 can adopt the following steps S301 to S304, and these steps are described in detail below.

[0081] S301, obtaining the passenger's flight date.

[0082] In the embodiment of the present application, when a passenger submits a lost and found request, the passenger's basic information (such as name, gender, age, etc.) and flight information (such as flight number, check-in time, boarding time, flight date, etc.) can also be queried at the same time. Based on this, the passenger's flight date can be obtained.

[0083] S302, obtaining the first picking-up date of the recorded picked-up object.

[0084] In the embodiment of the present application, for the found items that have been entered in the lost and found office, the corresponding first picking up date, that is, the date when the found items were picked up, can be recorded in advance.

[0085] S303: Determine, from the picked-up items, a first target picked-up item whose first picking-up date is later than the flight date.

[0086] In the embodiment of the present application, by comparing the flight date with the first pick-up date of the picked-up, the first target picked-up whose first pick-up date is later than the flight date can be determined. Of course, in some scenarios, the time period during which the first pick-up date is later than the flight date can also be limited, such as the first pick-up date being within ten days after the flight date.

[0087] S304, acquiring an object image of the first target object.

[0088] In the embodiment of the present application, after the first target object is determined, its corresponding object image can be obtained.

[0089] In a possible implementation, the Euclidean distance can be used to measure the similarity between the found object and the target lost object. Figure 4 , Figure 4 Another partial flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application. Figure 4 As shown, an embodiment of the present application provides a lost and found method based on multimodal learning, wherein step S40 "determines the target lost property from the found objects by matching the first image feature and the second image feature" can adopt the following steps S401 to S403, and these steps are described in detail below.

[0090] S401, calculating the Euclidean distance between the first image feature and the second image feature.

[0091] S402, determining the second target object with the smallest Euclidean distance among the objects.

[0092] S403: Push the picked-up image of the second target picked-up object to the passenger, so that the passenger can identify the target picked-up object from it.

[0093] Specifically, in an embodiment of the present application, the Euclidean distance between the first image feature and the second image feature can be calculated; then, at least one second target picked-up object with the smallest Euclidean distance is determined from the picked-up objects; finally, the picked-up image of the second target picked-up object is pushed to the passenger, and the passenger determines his target lost object from it.

[0094] In actual applications, it is sometimes difficult to accurately match lost items based solely on the description of the items (due to vague or general descriptions, such as a black backpack, a silver suitcase, etc.). Such descriptions will result in too many second target items being matched by the Euclidean distance, making it difficult for passengers to accurately locate the items.

[0095] See also Figure 5 , Figure 5 Another partial flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application. Figure 5 As shown, an embodiment of the present application provides a lost and found method based on multimodal learning, wherein step S40 "determines the target lost property from the found objects by matching the first image feature and the second image feature" may also include the following step S404, which is described in detail below.

[0096] S404, determining whether the number of the second target objects is greater than a preset number threshold; if not, executing step S403.

[0097] In the embodiment of the present application, a quantity threshold of the Euclidean distance matching can be preset to determine whether the number of the second target objects is greater than the quantity threshold; if the number of the second target objects is less than or equal to the quantity threshold, it means that the number of second target objects matched by the Euclidean distance is small, and then step S403 is executed.

[0098] In addition, in a possible implementation, when there are many second target objects matched by the Euclidean distance, a secondary match is performed using the external attributes of the objects. Figure 6 , Figure 6 Another partial flow chart of a lost and found method based on multimodal learning provided in an embodiment of the present application. Figure 6 As shown, the embodiment of the present application provides a lost and found method based on multimodal learning, wherein step S40 "determines the target lost object from the found objects by matching the first image feature and the second image feature" may also include the following steps S405 to S409, which are described in detail below:

[0099] S405, if yes, obtain the loss date of the target lost object and the movement location of the passenger in the airport, and obtain the second picking up date and picking up location of the second target picked up object, the movement location and the picking up location are both located in three-dimensional space.

[0100] In the embodiment of the present application, if the number of second target found items is greater than the above number threshold, it means that the second target found items matched by the Euclidean distance are more. When a passenger submits a lost and found request, the passenger may be required to enter the date of loss of the target lost item. Of course, the passenger's movement location in the airport (such as the check-in counter, security checkpoint, registration gate, etc.) can also be obtained based on the passenger's flight number.

[0101] In addition, for the found items that have been entered in the lost and found office, the corresponding second picking-up date, that is, the date when the found items were picked up, and the picking-up location point can be recorded in advance.

[0102] It should be noted that the movement position point and the pickup position point can be mapped into a three-dimensional space, including the corresponding longitude, latitude and floor, so as to generate the three-dimensional coordinates of the movement position point and the three-dimensional coordinates of the pickup position point.

[0103] S406, determining the time difference between the loss date and the second pick-up date, and the shortest distance between the movement position point and the pick-up position point.

[0104] In the embodiment of the present application, on the one hand, the time difference between the loss date and the second picking-up date can be calculated, and on the other hand, the straight-line distance between the moving position point and the picking-up position point in the three-dimensional space can be calculated as the shortest distance.

[0105] S407, calculating the matching probability between the second target found object and the target lost object according to the time difference and the shortest distance, wherein the matching probability is negatively correlated with the time difference, and the matching probability is negatively correlated with the shortest distance.

[0106] In the embodiment of the present application, the concept of time decay is introduced. As time goes by, the probability of the lost object being matched with the owner becomes smaller and smaller. The following formula (1) can be used to calculate the matching probability of the second target found object and the target lost object:

[0107] (1)

[0108] in, represents the matching probability, Indicates the shortest distance, Indicates the second pick-up date, Indicates the lost date, and 0.05 indicates the weight coefficient of time decay, which can be adjusted according to the actual scenario.

[0109] S408: Match the third target objects with the highest probability from the second target objects, and the number of the third target objects is the same as the number threshold.

[0110] In the embodiment of the present application, at least one third target picked-up with the highest matching probability can be selected from the second target picked-up according to the quantity threshold, and the number of the selected third target picked-up is the same as the quantity threshold. This greatly narrows the scope of lost items, allowing passengers to accurately and quickly locate lost items.

[0111] S409: Push the picked-up image of the third target picked-up object to the passenger, so that the passenger can identify the target picked-up object from it.

[0112] In the embodiment of the present application, at least one picked-up image of a third target picked-up object is pushed to the passenger, and the passenger determines his / her target lost object from the picked-up image.

[0113] Through the above description, a lost and found method based on multimodal learning provided in the embodiment of the present application has the following advantages: 1) Multi-dimensional prediction, not only focusing on the internal attributes of the object (type, color, etc.), but also focusing on the external attributes of the object (loss date, pick-up location, etc.), greatly increasing the accuracy of the algorithm; 2) saving manpower and reducing costs; 3) multimodal learning has strong scalability; 4) for the unstable trajectory of passengers in the airport, targeted rule correction can be carried out later.

[0114] A lost and found method based on multimodal learning provided in an embodiment of the present application is introduced above. A device for executing the lost and found method based on multimodal learning is introduced below.

[0115] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of a multimodal learning lost and found device provided in an embodiment of the present application. Figure 7 As shown, the multimodal learning lost and found device includes:

[0116] The multimodal learning module 10 is used to train the deep Boltzmann machine using the second description text and the lost object image of the calibrated lost object as training data in advance to obtain a multimodal learning network;

[0117] The lost property feature acquisition module 20 is used to acquire the first description text of the target lost property by the passenger, and extract the first text feature of the target lost property from the first description text; input the first text feature into the multimodal learning network to obtain the first image feature output by the multimodal learning network;

[0118] The lost object determination module 30 is used to obtain a picked-up object image and extract a second image feature of the picked-up object from the picked-up object image; and determine a target lost object from the picked-up objects by matching the first image feature and the second image feature.

[0119] In a possible implementation, the multimodal learning module 10 is specifically used for:

[0120] Construct a basic model, which includes a first deep Boltzmann machine and a second deep Boltzmann machine, and the first deep Boltzmann machine and the second deep Boltzmann machine establish a joint representation through a joint layer; determine the sample lost object used for this iterative training from the calibrated lost objects; obtain a second description text of the sample lost object, and extract a second text feature of the sample lost object from the second description text, and obtain a lost object image of the sample lost object, and extract a third image feature of the sample lost object from the lost object image; input the second text feature into the first deep Boltzmann machine, and input the third image feature into the second deep Boltzmann machine, iteratively update the weight parameters of the first deep Boltzmann machine, the second deep Boltzmann machine and the joint layer by calculating the probability distribution, and return to execute the step of determining the sample lost object used for this iterative training from the calibrated lost objects, until the preset end condition is met, and stop the iterative training; use the trained basic model as a multimodal learning network.

[0121] In a possible implementation, the lost object determining module 30 for acquiring the found object image is specifically configured to:

[0122] The flight date of the passenger is obtained; the first picking date of the recorded picked-up items is obtained; a first target picked-up item whose first picking date is later than the flight date is determined from the picked-up items; and a picked-up image of the first target picked-up item is obtained.

[0123] In a possible implementation, the lost object determination module 30 for determining the target lost object from the found objects by matching the first image feature and the second image feature is specifically configured to:

[0124] The Euclidean distance between the first image feature and the second image feature is calculated; a second target object with the smallest Euclidean distance among the objects is determined; and an object image of the second target object is pushed to the passenger so that the passenger can determine the target object from the image.

[0125] In a possible implementation, the lost object determining module 30, which is used to determine the target lost object from the found objects by matching the first image feature and the second image feature, is further used to:

[0126] It is determined whether the number of the second target objects is greater than a preset number threshold; if not, the step of pushing the picked-up image of the second target objects to the passenger is executed so that the passenger can determine the target objects from it.

[0127] In a possible implementation, the lost object determining module 30, which is used to determine the target lost object from the found objects by matching the first image feature and the second image feature, is further used to:

[0128] If so, obtain the loss date of the target lost object and the movement location of the passenger in the airport, and obtain the second pickup date and pickup location of the second target picked up object, both of which are located in the three-dimensional space; determine the time difference between the loss date and the second pickup date, and the shortest distance between the movement location and the pickup location; calculate the matching probability of the second target picked up object and the target lost object based on the time difference and the shortest distance, the matching probability is negatively correlated with the time difference, and the matching probability is negatively correlated with the shortest distance; match the third target picked up object with the highest probability from the second target picked up objects, and the number of the third target picked up objects is the same as the number threshold; push the picked up object image of the third target picked up object to the passenger, so that the passenger can determine the target picked up object from it.

[0129] It should be noted that the detailed functions of each module in the embodiment of the present application can be found in the corresponding public part of the above-mentioned multimodal learning lost and found method embodiment, which will not be repeated here.

[0130] The present application also provides an electronic device in an embodiment. Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0131] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0132] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a memory card, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0133] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any multimodal learning lost and found method provided in the embodiment of the present application.

[0134] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any multimodal learning lost and found method provided in the embodiment of the present application.

[0135] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0136] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0137] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0138] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A lost and found method based on multimodal learning, characterized in that: The lost and found method based on multimodal learning includes: Acquire a first description text of the target lost property by the passenger, and extract a first text feature of the target lost property from the first description text; Inputting the first text feature into a multimodal learning network to obtain a first image feature output by the multimodal learning network; wherein the multimodal learning network is obtained by training a deep Boltzmann machine in advance using the second description text of the calibrated lost object and the lost object image as training data; Acquire a picked-up object image, and extract a second image feature of the picked-up object from the picked-up object image; The target lost object is determined from the found objects by matching the first image feature and the second image feature.

2. The lost and found method based on multimodal learning according to claim 1, characterized in that: The process of training the deep Boltzmann machine using the second description text and the lost object image of the calibrated lost object as training data in advance to obtain the multimodal learning network includes: Constructing a basic model, wherein the basic model includes a first deep Boltzmann machine and a second deep Boltzmann machine, wherein the first deep Boltzmann machine and the second deep Boltzmann machine establish a joint representation through a joint layer; Determine the sample lost objects used for this iterative training from the calibrated lost objects; Acquire a second description text of the sample lost object, and extract a second text feature of the sample lost object from the second description text; and acquire a lost object image of the sample lost object, and extract a third image feature of the sample lost object from the lost object image; Input the second text feature into the first deep Boltzmann machine, and input the third image feature into the second deep Boltzmann machine, iteratively update the weight parameters of the first deep Boltzmann machine, the second deep Boltzmann machine and the joint layer by calculating the probability distribution, and return to the step of determining the sample lost items for this iterative training from the calibrated lost items, until the preset end condition is met, and then stop the iterative training; The trained basic model is used as the multimodal learning network.

3. The lost and found method based on multimodal learning according to claim 1, characterized in that: The step of obtaining a picked-up image of the picked-up object comprises: Get the flight date of the passenger; Get the first pickup date of the recorded pickup; Determine, from the picked-up items, a first target picked-up item whose first picking-up date is later than the flight date; Acquire the object image of the first target object.

4. The lost and found method based on multimodal learning according to claim 1, characterized in that: The step of determining the target lost object from the found objects by matching the first image feature and the second image feature comprises: calculating a Euclidean distance between the first image feature and the second image feature; Determine a second target object with the smallest Euclidean distance among the objects; The picked-up image of the second target picked-up object is pushed to the passenger, so that the passenger can identify the target picked-up object therefrom.

5. The lost and found method based on multimodal learning according to claim 4 is characterized in that: The determining the target lost object from the found objects by matching the first image feature and the second image feature also includes: Determining whether the number of the second target objects is greater than a preset number threshold; If not, the step of pushing the picked-up image of the second target picked-up object to the passenger is performed so that the passenger can identify the target picked-up object therefrom.

6. The lost and found method based on multimodal learning according to claim 5, characterized in that: The determining the target lost object from the found objects by matching the first image feature and the second image feature also includes: If yes, obtain the loss date of the target lost object and the movement location of the passenger in the airport, and obtain the second pick-up date and pick-up location of the second target found object, wherein the movement location and the pick-up location are both located in three-dimensional space; Determine the time difference between the lost date and the second picked-up date, and the shortest distance between the moving position point and the picked-up position point; Calculate the matching probability between the second target found object and the target lost object according to the time difference and the shortest distance, wherein the matching probability is negatively correlated with the time difference and the matching probability is negatively correlated with the shortest distance; Matching the third target objects with the highest probability from the second target objects, the number of the third target objects being the same as the number threshold; The picked-up image of the third target picked-up object is pushed to the passenger, so that the passenger can identify the target picked-up object therefrom.

7. A lost and found device based on multimodal learning, characterized in that: The lost and found device based on multimodal learning includes: A multimodal learning module, used to train a deep Boltzmann machine using a second description text and an image of the lost object as training data to obtain the multimodal learning network; A lost property feature acquisition module, used to acquire a first description text of a target lost property by a passenger, and extract a first text feature of the target lost property from the first description text; input the first text feature into the multimodal learning network to obtain a first image feature output by the multimodal learning network; The lost object determination module is used to obtain a found object image of the found object and extract a second image feature of the found object from the found object image; and determine the target lost object from the found object by matching the first image feature with the second image feature.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the lost and found method based on multimodal learning as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the lost and found method based on multimodal learning as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the lost and found method based on multimodal learning as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Lost article retrieval method, device and system and memory medium

    CN108183963A

  • A method and system for cross-modal retrieval

    CN109189968A

  • Article information transmission method and device

    CN115982356A

  • Article searching method and device, storage medium and electronic device

    CN117671002A