Passenger lost property searching method and system based on OCR (Optical Character Recognition) and readable medium

Abstract

The invention belongs to the technical field of computer vision and machine learning, and relates to an OCR (Optical Character Recognition)-based traveler lost property searching method and system and a readable medium, and the method comprises the following steps: obtaining a description text of a lost property, and extracting key information in the description text; inputting the key information and lost property images in a database into a multi-modal large model, and respectively extracting text features and image features of image-text alignment through an image encoder and a text encoder in the multi-modal large model; obtaining a matching value of the text feature and the image feature through a cross-modal matching algorithm; inputting the lost property image in the database into an OCR deep learning model, and extracting character information in the lost property image; correcting the matching value through the character information; and arranging the corrected matching values from high to low, and selecting a plurality of previous images as lost property searching results. The method can enhance the retrieval effect of railway lost items on images with text information such as names, brands and the like, such as books, medicines, foods and the like.

Description

Technical Field

[0001] The present invention relates to a method, system and readable medium for searching lost items of passengers based on OCR, and belongs to the technical field of computer vision and machine learning. Background Art

[0002] The number of lost items by railway passengers during their travels continues to increase. According to statistics, the number of lost items by railway passengers in 2023 will exceed 1.3 million pieces, while the current recovery rate of lost items on railways is only about 40%. Stations are under tremendous pressure in the storage, registration, search and management of lost items, which consumes a lot of manpower and material resources. The development of computer vision and artificial intelligence machine learning technology in the field of image and text retrieval has provided a new solution for finding lost items on railways.

[0003] In the prior art, for example, publication number CN116503843A discloses a method for automatically identifying and searching for lost objects, which intelligently identifies the lost objects and obtains the first characteristic information. When searching, the second characteristic information of the input object is obtained, and the second characteristic information is matched with the first characteristic information to find the target lost object. The first characteristic information is information such as the image of the lost object, and the second characteristic information is the description of the lost object by the owner. It can be seen that in the prior art, a deep learning network can be used to automatically search for lost objects based on the description of the owner. However, it also has the following problems: First, the original description language is natural language, and sometimes the description is very detailed, so the algorithm cannot automatically identify useful information; Second, when identifying lost items, judgments are usually made based on some detailed information, such as the text on a schoolbag, etc. However, in existing models, images are judged as a whole, such as color, size, etc. There may be more than one lost item of the same color and size, making it impossible to make a judgment. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to propose an OCR-based passenger lost property search method, system and readable medium, which can enhance the retrieval effect of railway lost property images with text information such as names and brands such as books, medicines, and foods.

[0005] To achieve the above-mentioned purpose, the present invention proposes the following technical scheme: a method for finding lost property of passengers based on OCR, comprising the following steps: obtaining a description text of the lost property and extracting key information from the description text; inputting the key information and the lost property image in the database into a multimodal large model, and extracting the text features and image features of the image and text alignment through the image encoder and the text encoder therein respectively; obtaining the matching values ​​of the text features and the image features through a cross-modal matching algorithm; inputting the lost property image in the database into an OCR deep learning model, and extracting the text information in the lost property image; correcting the matching value through the text information; arranging the corrected matching values ​​from high to low, and selecting the first several images as the lost property search results.

[0006] Furthermore, the method for extracting key information from the description text is as follows: the detailed description in the original user information is subjected to a large language model to extract key information; the refined description is directly sent to the multimodal large model.

[0007] Furthermore, the detailed description is a description in which the number of words in the original user information is greater than a threshold; the refined description is a description in which the number of words in the original user information is less than a threshold and a variety of different parts of speech information are detected by a word segmentation algorithm.

[0008] Furthermore, the matching values ​​of the text features and the image features are obtained through a cross-modal matching algorithm.

[0009] Furthermore, the cross-modal matching algorithm is a cosine similarity algorithm.

[0010] Furthermore, the text information in the lost object image is extracted; and the method for correcting the matching value by using the text is as follows: detecting whether there is text information in the image, and if so, extracting it by OCR; Generate a continuous text from the text information; determine whether the continuous text and the text feature have the same content, and if so, calculate the additional points for the matching value based on the same content, so as to correct the matching value.

[0011] Furthermore, it is determined whether there is the same content between the continuous text and the text feature. If so, the method for calculating the additional points for the matching value based on the same content is as follows: two consecutive characters in the continuous text are regarded as a block and slide in sequence to see whether continuous identical text appears in the key information describing the text. If so, the image-text matching value is increased.

[0012] Furthermore, the calculation formula for the additional points is:

[0013] in, It's a plus. is the weighting coefficient, is the highest weighted value, is the number of identical characters that occur.

[0014] The present invention also discloses a passenger lost property search system based on OCR, comprising: a description text acquisition module, used to obtain a description text of the lost property and extract key information from the description text; a multimodal large model module, used to input the key information and the lost property image in the database into the multimodal large model, and extract the text features and image features of the image and text alignment through the image encoder and the text encoder therein respectively; a matching value calculation module, used to obtain the matching values ​​of the text features and the image features through a cross-modal matching algorithm; a matching value correction module, used to input the lost property image in the database into the OCR deep learning model, and extract the text information in the lost property image; and correct the matching value through the text information; and a result output module, used to arrange the corrected matching values ​​from high to low, and select the first several images as the lost property search results.

[0015] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement any of the above-mentioned OCR-based passenger lost property search methods.

[0016] The technical solution of the present invention has at least the following technical effects or advantages: the present invention is a large-model image and text retrieval enhancement method based on OCR for railway lost objects. In deployment scenarios with limited hardware resources, it can further assist in enhancing the retrieval effect of images with text information such as names and brands, such as books, medicines, and foods. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of a method for finding lost property of passengers based on OCR in one embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail through specific embodiments. However, it should be understood that the provision of specific embodiments is only for a better understanding of the present invention, and they should not be understood as limitations of the present invention. In the description of the present invention, it should be understood that the terms used are only for the purpose of description and cannot be understood as indicating or implying relative importance.

[0019] In order to solve the problems existing in the prior art that the owner speaks natural language and sometimes describes it very carefully, and the algorithm cannot automatically recognize useful information; when identifying lost objects, judgment is usually made based on some detailed information, such as text on a school bag, but in the existing model, the image is judged from the whole, such as color, size, etc., there may be more than one lost object of the same color and size, and it is impossible to judge, etc., the present invention proposes a method, system and readable medium for searching lost objects of passengers based on OCR, which can recognize important information in natural language through a large language model, facilitate query speedup, obtain text information on the lost object through an OCR (optical character recognition) model, so as to facilitate the search of school bags, books, medicines, foods, etc., making this type of lost objects easier to find.

[0020] Large language models can efficiently extract key information from text through prompt engineering. By designing effective input prompts, these models can be guided to generate the desired output. This approach takes advantage of the powerful understanding of natural language by pre-trained language models and can be used for a variety of tasks, including but not limited to information extraction, question-answering systems, summary generation, etc. Multimodal large models can obtain feature representations of images and text through explicit embedding mechanisms. This mechanism allows the model to convert data from different modalities into representations in a unified vector space, allowing for effective cross-modal understanding and processing. Dedicated OCR (Optical Character Recognition) models are deep learning models specifically designed to extract text information from images. These models are usually trained with a large amount of annotated data, can handle text in a variety of different fonts, sizes, and arrangements, and have high recognition accuracy in complex backgrounds.

[0021] With the rapid development of artificial intelligence technology, the organic combination of text and image information can achieve accurate matching of text information and image information of lost items on railways, improve the recovery rate of lost items on railways, achieve cost reduction and efficiency improvement in the field of railway passenger transportation, and improve the service quality and management level of railway passenger transportation. The scheme of the present invention is described in detail below with reference to the accompanying drawings through embodiments.

[0022] Embodiment 1 This embodiment discloses a method for finding lost items of passengers based on OCR, comprising the following steps: S1 obtains a description text of the lost object and extracts key information from the description text.

[0023] The original description text is extracted through the large language model through the hint engineering to extract key information. That is, the rich corpus knowledge of the large language model is used to extract key information from the item description to form a simplified text.

[0024] The method for extracting key information from the description text is as follows: the detailed description in the original user information is extracted through a large language model; the refined description is directly sent to the multimodal large model.

[0025] Specifically, for example, key information extraction is as follows: User information A: "Hello, I am a passenger on the D22 train. My name is Wei. During the journey, I accidentally lost a cream-colored handbag. I am very anxious now and hope to get your help. The incident happened around 15:00 this afternoon. I may have placed the handbag on the seat of the carriage or on the luggage rack nearby. Due to the bumps during the train or my movement in other carriages, the handbag may have slipped from its original place or been taken by others by mistake. The handbag is cream-colored, the shoulder straps are black, and there are many letters of 'PANDA' printed on the surface. The bag is not particularly large. I really hope to find this handbag as soon as possible. Does the station have any relevant lost and found services? Or can I look for the lost property at the next station or the terminal where the train stops? Please help me keep an eye out. Thank you very much!" Key Extracted Information A: "A cream-colored handbag with a black strap and multiple letters of 'PANDA' printed on the surface." User information B: "A yellow plush hat." Key extracted information B: "A yellow plush hat." Because some of the original user information is a relatively detailed description, such as user information A, it needs to go through a large language model to extract key information; some is a relatively refined description, such as user information B, which can be directly sent to the next step of the multimodal large model and save running time.

[0026] A detailed description is a description whose original user information has more words than the threshold; a refined description is a description whose original user information has less words than the threshold, and a description in which a variety of different parts of speech information are detected by the word segmentation algorithm. That is, if the original user information has more words than s, it needs to be extracted through a large language model; otherwise, if the original user information has less words than s, and the word segmentation algorithm detects that it contains nouns, adjectives, etc., it does not need to be extracted through a large language model, and can be directly sent to the next step of the multimodal large model.

[0027] S2 inputs the key information and the lost object images in the database into the multimodal large model, and extracts the text features and image features of the image-text alignment through the image encoder and text encoder respectively.

[0028] S3 obtains the matching values ​​of text features and image features through a cross-modal matching algorithm.

[0029] The matching values ​​of text features and image features are obtained through cross-modal matching algorithms such as cosine similarity algorithm.

[0030] S4 inputs the lost object image in the database into the OCR deep learning model to extract the text information in the lost object image; and corrects the matching value based on the text information.

[0031] Extract text information from the lost object image; the method for correcting the matching value through text is: Detect whether there is text information in the image, and if so, extract it through OCR; Generate a continuous text from text information; Determine whether there is the same content between the continuous text and the text feature. If so, calculate the additional points for the matching value based on the same content, so as to correct the matching value.

[0032] The method of judging whether there is the same content between the continuous text and the text feature, and if so, calculating the additional points for the matching value based on the same content is as follows: treat two consecutive characters in the continuous text as a block and slide them one by one to see whether the same characters appear in the key information of the description text. If so, the additional points are given to the image-text matching value. The calculation formula for the additional points is:

[0033] in, It's a plus. is the weighting coefficient, is the highest weighted value, is the number of identical characters that occur.

[0034] S5 arranges the corrected matching values ​​from high to low, and selects the first several images as the lost object search results.

[0035] For each original text description, the top k images matched are output as image-text retrieval results according to the image-text matching values ​​from high to low.

[0036] This embodiment is based on the OCR large-model image and text retrieval enhancement method for railway lost objects. In deployment scenarios with limited hardware resources, it can further assist in enhancing the retrieval effect of images with text information such as names and brands, such as books, medicines, and foods.

[0037] Embodiment 2 Based on the same inventive concept, this embodiment discloses a passenger lost property search system based on OCR, including: A description text acquisition module is used to obtain the description text of the lost object and extract key information from the description text; The multimodal large model module is used to input key information and the lost object image in the database into the multimodal large model, and extract the text features and image features of the image-text alignment through the image encoder and text encoder therein; A matching value calculation module is used to obtain matching values ​​of text features and image features through a cross-modal matching algorithm; The matching value correction module is used to input the lost object image in the database into the OCR deep learning model, extract the text information in the lost object image, and correct the matching value according to the text information; The result output module is used to arrange the corrected matching values ​​from high to low and select the first several images as the lost object search results.

[0038] Embodiment 3 Based on the same inventive concept, this embodiment discloses a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement any of the above-mentioned OCR-based passenger lost property search methods.

[0039] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0040] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0041] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0042] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not deviate from the spirit and scope of the present invention should be included in the protection scope of the claims of the present invention. The above content is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for finding lost property of passengers based on OCR, characterized in that: The following steps are involved: Obtaining a description text of the lost object and extracting key information from the description text; Input the key information and the lost item image in the database into the multimodal large model, and extract the text features and image features of the image-text alignment through the image encoder and the text encoder therein respectively; Through the cross-modal matching algorithm, the matching values ​​of text features and image features are obtained; Inputting the lost object image in the database into the OCR deep learning model to extract the text information in the lost object image; and correcting the matching value according to the text information; Arrange the corrected matching values ​​from high to low, and select the first several images as the lost object search results.

2. The method for finding lost property of passengers based on OCR as claimed in claim 1, characterized in that: The method for extracting key information from the description text is as follows: the detailed description in the original user information is subjected to a large language model to extract key information; the refined description is directly sent to the multimodal large model.

3. The OCR-based passenger lost property search method according to claim 2, characterized in that: The detailed description is a description in which the number of words in the original user information is greater than a threshold; the refined description is a description in which the number of words in the original user information is less than a threshold and a variety of different parts of speech information are detected through a word segmentation algorithm.

4. The method for finding lost property of passengers based on OCR as claimed in claim 1, characterized in that: The matching values ​​of the text features and the image features are obtained through a cross-modal matching algorithm.

5. The method for finding lost property of passengers based on OCR as claimed in claim 4, characterized in that: The cross-modal matching algorithm is a cosine similarity algorithm.

6. The method for finding lost property of passengers based on OCR as claimed in claim 1, characterized in that: The method of extracting text information from the lost object image and correcting the matching value by using the text is: Detect whether there is text information in the image, and if so, extract it through OCR; Generate a continuous text from the text information; It is determined whether the continuous text and the text feature have the same content. If so, a bonus for the matching value is calculated based on the same content, thereby correcting the matching value.

7. The method for finding lost property of passengers based on OCR as claimed in claim 6, characterized in that: The method for judging whether there is the same content between the continuous text and the text feature, and if so, calculating the additional points for the matching value based on the same content is as follows: treating two consecutive characters in the continuous text as a block and sliding them in sequence to see whether continuous identical text appears in the key information describing the text, and if so, adding points to the image-text matching value.

8. The method for finding lost property of passengers based on OCR as claimed in claim 7, characterized in that: The calculation formula for the additional points is: in, It's a plus. is the weighting coefficient, is the highest weighted value, is the number of identical characters that occur.

9. A passenger lost property search system based on OCR, characterized in that: include: A description text acquisition module is used to acquire the description text of the lost object and extract key information from the description text; The multimodal large model module is used to input the key information and the lost object image in the database into the multimodal large model, and extract the text features and image features of the image-text alignment through the image encoder and the text encoder therein respectively; A matching value calculation module is used to obtain matching values ​​of text features and image features through a cross-modal matching algorithm; A matching value correction module is used to input the lost object image in the database into the OCR deep learning model to extract text information in the lost object image; Modifying the matching value by using the text information; The result output module is used to arrange the corrected matching values ​​from high to low and select the first several images as the lost object search results.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the OCR-based method for finding lost property of passengers as described in any one of claims 1 to 7.

Images