System

Abstract

A system is provided.SOLUTION: A system includes a means for photographing an image of baggage by a user terminal, a means for analyzing the image photographed by a server and detecting a prohibited article or a restricted article, and a means for notifying a user of a detection result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] During security and customs inspections when traveling, rules regarding carry-on and checked baggage vary by country and airline, which can easily lead to problems. Specifically, travelers often enter inspections without knowing and have their belongings confiscated, or the inspection process takes longer than necessary, resulting in long lines. Furthermore, quarantine and customs prohibitions, restrictions, and taxation rules vary from country to country, which can be confusing when entering the country and poses a risk of smuggling. These problems need to be solved efficiently. [Means for solving the problem]

[0005] This invention proposes a system that includes a means for taking images of carried items using a user terminal, a means for a server to analyze the captured images and detect prohibited or restricted items, and a means for notifying the user of the detection results. The system also includes a means for the server to compare the analysis results with travel information linked to the passport and confirm the detected prohibited or restricted items. Furthermore, a means is provided for the server to automatically generate customs declaration documents based on a list of carried items and display them on the user terminal. Finally, the system includes a means for the user terminal to capture audio and video data and send it to the server, and a means for the server to analyze the received data and detect suspicious behavior and speech, and then notify the security department. This aims to improve the efficiency of security and customs inspections during travel and reduce problems.

[0006] A "user terminal" is an electronic device carried by a traveler and used for advance preparations before departure and during immigration procedures.

[0007] A "server" is a computer system that receives data sent from a user terminal, analyzes and processes it, and provides the results to the user terminal.

[0008] "Images of belongings" are photographs or video data of luggage owned by the user.

[0009] The "means for taking an image" is a method for taking an image of a carried item using the camera function of the user terminal.

[0010] "Means for analyzing images" refers to a method in which the server processes the image data it receives using AI technology, etc., to recognize and classify items.

[0011] "Prohibited items" are items that are prohibited from being carried on board or checked in according to the regulations of the destination country or airline.

[0012] "Restricted items" are items that may only be carried or checked under certain conditions.

[0013] "Customs declaration form" means a declaration required by customs in the destination country, which contains information about the items you are carrying.

[0014] "Smuggling" is the act of illegally bringing goods into a country by unlawful means.

[0015] "Audio and video data" refers to audio recordings and video footage captured by the user terminal.

[0016] "Suspicious behavior or speech" refers to behavior or speech that differs from that of a normal traveler and may suggest smuggling or illegal activity.

[0017] A "security notification" is the act of sending a warning to responsible agencies such as airport security when the server detects suspicious behavior or speech as a result of its analysis. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The embodiment of the present invention will be described with a focus on a user terminal, a server, and data exchange between them.

[0040] Advance preparation before departure and image recognition

[0041] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[0042] Device: The captured images are temporarily stored on the device, which then uploads them to the server.

[0043] Server: The server analyzes the received image data. During the analysis, AI technology is used to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item. The result of the determination is sent to the user's device.

[0044] Travel information collation and notification

[0045] Server: After obtaining the image analysis results, the server checks the user's passport information. Specifically, it compares the user's belongings with the destination country's regulations and airline regulations to see if they violate any rules. If prohibited or restricted items are detected, the server notifies the user.

[0046] Terminal: The user terminal displays the notification received from the server and prompts the user to take necessary measures, such as changing prohibited items into checked baggage.

[0047] Auto-generation of customs declaration documents

[0048] Server: The server automatically generates the necessary customs declaration forms based on the list of recognized items, including determining whether the items require declaration.

[0049] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed to the user on the terminal, allowing the user to follow the instructions and prepare for a smooth customs process.

[0050] Smuggling detection at the point of entry

[0051] Device: During immigration procedures, the user's device captures audio and video data and transmits it to a server in real time, allowing the user's actions and behavior during the procedure to be monitored.

[0052] Server: The server analyzes the received audio and video data to detect suspicious behavior or actions, such as when a user makes contradictory statements or exhibits specific non-verbal behavior.

[0053] Server: When suspicious activity is detected, the server notifies security personnel in real time, allowing for any necessary further investigations or responses to be taken promptly.

[0054] Specific examples

[0055] Example 1: Image recognition before departure

[0056] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[0057] Example 2: Automatic generation of customs declaration documents

[0058] When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. By presenting this form at the customs office, the user can smoothly proceed with customs clearance.

[0059] Example 3: Detecting contraband

[0060] If a user behaves suspiciously during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security, and additional measures are taken.

[0061] This system will reduce immigration issues and significantly improve the efficiency of security and customs inspections.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] User: Takes pictures of their belongings using a smartphone or dedicated device. The user takes pictures of all items in turn and records the necessary information.

[0065] Step 2:

[0066] Device: Temporarily stores captured images. Once all items have been photographed, the image data will be ready to upload.

[0067] Step 3:

[0068] On the device: Upload the captured image to the server. Display a notification to the user when the upload is complete.

[0069] Step 4:

[0070] Server: Analyzes the received image data. Using AI technology, it recognizes and classifies the objects in each image.

[0071] Step 5:

[0072] Server: Compares the analysis results with the destination country's regulations and airline rules to identify prohibited or restricted items.

[0073] Step 6:

[0074] Server: If prohibited or restricted items are detected, it generates a list of them and creates notification data.

[0075] Step 7:

[0076] Terminal: Receives notification data sent from the server and displays it to the user. The user checks the displayed information and takes necessary measures.

[0077] Step 8:

[0078] Server: Based on the list of recognized items, a customs declaration form is automatically generated, listing the items that must be declared.

[0079] Step 9:

[0080] Server: Sends automatically generated customs declaration documents to the user's terminal.

[0081] Step 10:

[0082] Terminal: Displays the customs declaration documents sent to the user's terminal. The user checks the contents of the documents and prepares for customs procedures.

[0083] Step 11:

[0084] Terminal: During immigration procedures, audio and video data is captured and transmitted to a server in real time.

[0085] Step 12:

[0086] Server: Analyzes received audio and video data to detect suspicious behavior and speech, including non-verbal behavior and contradictory statements.

[0087] Step 13:

[0088] Server: If suspicious activity is detected, it notifies security personnel in real time and sends the analysis results as additional information.

[0089] Step 14:

[0090] Security personnel: Receive notifications from the server and promptly carry out any necessary additional checks or actions.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Conventional immigration control systems require users to perform numerous manual tasks when handling carry-on baggage and customs declaration procedures, resulting in inefficiency and a high likelihood of errors. Furthermore, detecting smuggling during immigration inspections relies on human resources, making thorough monitoring difficult. Therefore, there is a need for a system that can improve efficiency and prevent fraud in each process of carry-on baggage management, customs declaration, and immigration inspection.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes means for taking images of carried items using a user terminal, means for temporarily saving the images taken by the terminal, means for uploading the images saved by the terminal to the server, means for analyzing the image data received by the server using AI technology to recognize and classify the items, means for determining whether the items fall under prohibited or restricted items, and means for transmitting the determination result to the user terminal. This automates the management of carried items, reduces the burden on users, and enables the prevention of fraudulent activities and the efficiency of immigration procedures.

[0096] A "user terminal" is an electronic device such as a mobile information terminal or a smartphone that is intended to be operated by a user.

[0097] "Personal effects" refers to all items that a user carries with them when traveling or moving.

[0098] The "means for taking an image" is a function for taking an image of a carried item using a camera function provided in the user terminal.

[0099] The "temporary storage means" is a function for storing captured images in a temporary storage device within the user terminal.

[0100] A "server" is a remote data processing device that receives and processes data sent from a user terminal.

[0101] "Uploading means" is a function for transmitting data from a user terminal to a server.

[0102] "Means for analyzing image data using AI technology" refers to a processing function that uses AI technology on a server to identify and classify objects contained in images.

[0103] "Means for recognizing and classifying items" refers to a function that identifies and classifies the name and type of an item from image data analyzed by AI.

[0104] "Means for determining whether a recognized item is a prohibited or restricted item" is a function that determines whether the recognized item violates the laws and regulations of the destination country.

[0105] The "means for transmitting the determination result to the user terminal and notifying the user" is a function for transmitting the determination result to the user terminal and displaying a notification to the user.

[0106] "Travel information" refers to all information related to a user's travel, such as passport information and airline ticket information.

[0107] "Means of matching" refers to the function of comparing the image analysis results with travel information to check for matches or violations.

[0108] The "means for automatically generating customs declaration documents" is a function that automatically creates the necessary customs declaration documents based on the analyzed carry-on item data.

[0109] The "means for displaying the automatically generated customs declaration document on the user terminal" is a function for transmitting the generated customs declaration document to the user terminal so that the user can view it on the terminal.

[0110] The present invention relates to a system for improving the efficiency of carry-on baggage management and customs declaration procedures at the time of immigration and for strengthening smuggling detection. Specific embodiments of the system are described below.

[0111] Preparation before departure and image recognition

[0112] User: Before leaving the country, the user takes pictures of their belongings using their smartphone or a dedicated device. This is done using the smartphone's camera function. For example, the user can take pictures of each item they put in their suitcase, obtaining image data of all of their belongings.

[0113] On the device: The captured images are saved in a temporary folder (local storage) on the user's device. They are then automatically uploaded from the device to the server. This upload process is carried out via a dedicated application on the device.

[0114] Server: The server analyzes the received image data using AI technology. Specifically, it uses image analysis libraries such as TENSORFLOW (registered trademark) and OpenCV to recognize and classify objects. For example, objects in the image are recognized as "lighter," "bottle," "book," etc.

[0115] Server: Based on the analysis results, the server determines whether the item is a prohibited or restricted item. This involves checking it against a list of prohibited items stored on the server. The result of the determination is then sent to the user's device. The user is notified and can take the necessary measures accordingly.

[0116] Travel information collation and notification

[0117] Server: After obtaining the image analysis results, the server checks the user's passport information (electronic passport data) and travel information to ensure that the belongings comply with the regulations of the destination country and airline.

[0118] Server: If a prohibited or restricted item is detected, the server sends that information to the user's device and notifies the user of the appropriate action to take, such as "Lighters are not permitted on board. Please put them in checked baggage."

[0119] Auto-generation of customs declaration documents

[0120] Server: The server automatically generates the necessary customs forms based on the list of recognized items, including determining whether the items require declaration.

[0121] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed in a PDF viewer or dedicated application, allowing the user to follow the instructions and prepare for smooth customs procedures.

[0122] Smuggling detection at the point of entry

[0123] Terminal: During immigration control, the user's terminal captures audio and video data and transmits it to the server in real time using the terminal's built-in camera and microphone.

[0124] Server: The server analyzes the received audio and video data to detect suspicious behavior and speech. For example, it uses voice recognition and facial recognition systems to analyze the user's speech and non-verbal behavior (e.g., facial expressions and body movements).

[0125] Server: If any suspicious activity is detected, the server immediately notifies security personnel and prompts them to take any necessary further investigations or actions, thereby monitoring potential smuggling activity in real time.

[0126] Specific examples

[0127] Example 1: Before leaving the country, a user takes a picture of their baggage with their smartphone, and the device sends the picture to a server. If the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as checked baggage."

[0128] Example 2: When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. The user then presents the form at the customs office and the customs clearance process goes smoothly.

[0129] Example 3: If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security, and additional measures are taken.

[0130] This system will reduce immigration issues and significantly improve the efficiency of security and customs inspections.

[0131] Prompt Sentence Examples

[0132] Example: "How can I take a picture of my baggage before leaving the country and upload it to a server, so that AI can recognize and classify the items and notify the user if any prohibited or restricted items are detected?"

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] User: The user uses a smartphone or dedicated device to take pictures of each item in their baggage. Specifically, the user launches a camera app and takes pictures of each item of baggage.

[0136] Input: The physical state of the baggage.

[0137] Output: Digital image data.

[0138] Step 2:

[0139] Device: The captured image is automatically saved to a temporary folder on the user's device.

[0140] Input: Images taken with a camera app on a smartphone or dedicated device.

[0141] Output: Image file saved in the device's local storage.

[0142] Step 3:

[0143] On the device: To upload the saved image to the server, send the image data via the API. Specifically, send the image data to the server using an HTTP request.

[0144] Input: An image file stored in local storage.

[0145] Output: Image data sent to the server.

[0146] Step 4:

[0147] Server: The server analyzes the received image data using AI technology, for example, using image analysis libraries such as TensorFlow or OpenCV, and applies a model to recognize objects in the image.

[0148] Input: Image data sent to the server.

[0149] Output: Identified item information as a result of analysis (e.g. lighter, plastic bottle, book, etc.).

[0150] Step 5:

[0151] Server: Based on the analysis results, it determines whether the item is a prohibited or restricted item, and in this process, it checks against the prohibited items list.

[0152] Input: Identified item information as a result of analysis.

[0153] Output: Judgment result (e.g. prohibited item, restricted item, permitted item).

[0154] Step 6:

[0155] Server: Sends the judgment result to the user device. Specifically, it generates JSON data in the form of a notification and sends it to the user device.

[0156] Input: The result of the test.

[0157] Output: Notification data sent to the user device.

[0158] Step 7:

[0159] Terminal: The user terminal analyzes the received notification data and displays it to the user. Specifically, it uses a pop-up notification or an alert dialog to alert the user.

[0160] Input: Notification data sent by the server.

[0161] Output: The notification displayed on the user's screen.

[0162] Step 8:

[0163] Server: Compares the user's passport and travel information with the image analysis results, specifically by accessing a database and comparing them with the destination country's regulations and airline regulations.

[0164] Input: Passport information, travel information, and image analysis results.

[0165] Output: Verification data as a result of the match (e.g., whether or not there is a violation).

[0166] Step 9:

[0167] Server: If prohibited or restricted items are detected based on the matching results, the server notifies the user of this information.

[0168] Input: Verification data as a match result.

[0169] Output: The alert data sent to the user terminal.

[0170] Step 10:

[0171] Terminal: The user's terminal analyzes the received warning data and notifies the user of countermeasures. For example, it displays specific instructions such as "Lighters are prohibited from being carried on board. Please put them in checked baggage."

[0172] Input: The alert data sent by the server.

[0173] Output: The instructions displayed on the user's screen.

[0174] Step 11:

[0175] Server: Automatically generates customs declaration documents based on the list of recognized carry-on items. Specifically, it automatically fills in the necessary information in a template and generates the documents in PDF format.

[0176] Input: A list of recognized items.

[0177] Output: Auto-generated customs declaration form (PDF).

[0178] Step 12:

[0179] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed for the user to view.

[0180] Input: Auto-generated customs declaration form (PDF).

[0181] Output: Customs declaration document displayed on user's terminal.

[0182] Step 13:

[0183] Terminal: During immigration control, the user terminal captures audio and video data and transmits it to the server in real time using the built-in camera and microphone.

[0184] Input: Audio and video data from immigration inspection.

[0185] Output: Audio and video data sent to the server.

[0186] Step 14:

[0187] Server: The server analyzes the received audio and video data and detects suspicious behavior or speech. Specifically, it uses voice recognition and facial recognition systems.

[0188] Input: Audio and video data sent to the server.

[0189] Output: Evaluation data of behavior and speech as the analysis result.

[0190] Step 15:

[0191] Server: If suspicious activity is detected, the server notifies security personnel in real time, facilitating further investigation and response.

[0192] Input: Evaluation data of behavior and speech as the analysis results.

[0193] Output: Alert data sent to security personnel.

[0194] This processing step reduces immigration issues and significantly improves the efficiency of security and customs inspections.

[0195] (Application example 1)

[0196] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] When traveling internationally, it is cumbersome to check whether your belongings comply with regulations before leaving the country, and it takes time to fill out customs declaration forms. Furthermore, there is a lack of means to quickly identify travelers exhibiting suspicious behavior during immigration inspections, making it difficult to ensure security. There is a need for a solution to these problems, making immigration procedures more efficient and improving security.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0199] In this invention, the server includes means for taking images of carried items using a user terminal, means for analyzing the images taken by the server to detect prohibited or restricted items, means for notifying the user of the detection results, means for automatically generating customs declaration forms and displaying them on the user terminal, means for capturing audio and video data during immigration inspection and transmitting it to the server in real time, means for analyzing the audio and video data in the server to detect suspicious behavior, and means for notifying security personnel when suspicious behavior is detected. This enables quick confirmation of carried items before departure, automates the creation of customs declaration forms, and strengthens security inspections at the time of entry.

[0200] A "user terminal" is an electronic device that is carried and operated by a user, and is a device for taking images of carried items and receiving notifications.

[0201] A "server" is a computer system connected to a network that stores, analyzes, and processes data.

[0202] "Personal effects" are items that a user carries with them when traveling or going out.

[0203] The "means for taking an image" is a mechanism for taking a photo of a carried item using the camera function installed in the user terminal.

[0204] The "means for analyzing images" refers to algorithms or software for processing the image data received by the server and detecting specific items.

[0205] "Prohibited items" are items that are prohibited from being possessed or brought into the country under laws and regulations.

[0206] "Restricted items" are items whose possession or import is restricted based on permits or conditions.

[0207] "Means for notifying" refers to a technology for transmitting information from a server to a user terminal in real time and informing the user.

[0208] "Customs Declaration Form" means an official document used to declare possession of goods to customs.

[0209] "Automatic generation means" refers to a system that uses a server algorithm to automatically generate the necessary documents.

[0210] "Means for capturing audio and video data" refers to the process of acquiring audio and video using a microphone and camera installed on the user terminal.

[0211] "Means for transmitting in real time" refers to Internet communication technology for instantly transmitting data to a server.

[0212] "Means for detecting suspicious behavior" refers to technology that automatically detects suspicious behavior and speech based on audio and video data on the server.

[0213] "Means for notifying security personnel" refers to a system for sending warning information to security personnel when the server detects suspicious activity.

[0214] The present invention provides a system for improving the efficiency of baggage management before departure and security checks at the time of entry through data exchange between a user terminal and a server. Specific embodiments of the system are described below.

[0215] Baggage management before departure

[0216] 1. Taking pictures using a user device

[0217] A user takes a picture of their belongings with a user terminal such as a smartphone, and image data of the baggage is acquired using the camera of the smartphone.

[0218] 2. Sending image data

[0219] The captured image data is uploaded from the user's device to a server via an internet connection, and the data is sent in real time.

[0220] 3. Server Image Analysis

[0221] The server analyzes the received image data using AI technology. Specifically, it uses an object recognition algorithm to identify items in the image and detect prohibited or restricted items. For this, image analysis software such as OpenCV or TensorFlow can be used.

[0222] 4. Notification of test results

[0223] The analysis results are sent to the user's device, where the user can check whether or not there are any prohibited or restricted items. If the user accidentally carries a prohibited item, the device will notify the user in real time.

[0224] 5. Automatic generation of customs declaration documents

[0225] The server automatically generates customs declaration forms based on the analyzed list of carried items, and the generated forms are sent to the user's terminal, where the user can view and print them.

[0226] Security checks at entry

[0227] 1. Audio and Video Capture

[0228] During immigration inspection, the user's device (smartphone) captures audio and video data in real time and transmits it to a server using the smartphone's microphone and camera.

[0229] 2. Real-time data analysis

[0230] The server analyzes the received audio and video data, using natural language processing technology for audio analysis and object detection and motion analysis technology for video analysis, to detect suspicious behavior or contradictory statements in real time.

[0231] 3. Security Notice

[0232] If any suspicious activity is detected, the server immediately notifies security personnel, allowing any necessary additional testing or response to be carried out promptly.

[0233] Examples of specific examples and prompts

[0234] Example 1: Image recognition before departure

[0235] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[0236] Example 2: Security checks at entry

[0237] If a user behaves suspiciously during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security personnel, who then take additional action.

[0238] Prompt for the generative AI model (luggage analysis example):

[0239] "Analyze the image to determine if it contains any illegal items. The image data is contained in the next argument, image_data."

[0240] The above embodiment is expected to improve the efficiency of immigration procedures and security.

[0241] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0242] Step 1:

[0243] A user takes a photo of their belongings with a user terminal such as a smartphone. The captured image data is acquired using the smartphone's camera. The input is the image data captured by the user's operation, and the output is the image data temporarily saved on the terminal.

[0244] Step 2:

[0245] The device uploads the captured image data to a server. The data is transmitted in real time via an internet connection. The input is the captured image data, and the output is the image data transmitted to the server.

[0246] Step 3:

[0247] The server analyzes the received image data using AI technology. Specifically, it uses object recognition algorithms (e.g., OpenCV or TensorFlow) to identify items in the image and detect prohibited or restricted items. The input is the image data sent to the server, and the output is the analysis results.

[0248] Step 4:

[0249] The server sends a notification to the user's device based on the analysis results. The notification includes details of the prohibited or restricted items detected. The input is the result data from the AI ​​analysis, and the output is a notification message sent to the user's device.

[0250] Step 5:

[0251] The server automatically generates customs declaration forms based on the analyzed list of carried items, including determining which items should be declared. The input is the list of items based on the analysis results, and the output is the generated customs declaration form.

[0252] Step 6:

[0253] The server sends the automatically generated customs declaration form to the user's terminal, where the user can check and print it. The input is the generated customs declaration form, and the output is the declaration form sent to the user's terminal.

[0254] Step 7:

[0255] During immigration control, the user's device (smartphone) captures audio and video data and transmits it to the server in real time. The data is acquired using the smartphone's microphone and camera. The input is the audio and video data captured at the immigration control site, and the output is the audio and video data transmitted to the server.

[0256] Step 8:

[0257] The server analyzes the received audio and video data. It uses natural language processing technology for audio analysis and object detection and motion analysis technology for video analysis. The input is the captured audio and video data, and the output is behavioral judgment data as the analysis result.

[0258] Step 9:

[0259] Based on the analysis results, the server determines whether there is any suspicious behavior or contradictory statements, and notifies security personnel if any suspicious behavior is detected. The input is behavior determination data, and the output is a notification message to the security personnel.

[0260] These steps ensure efficient and effective baggage checks before departure, automatic generation of customs declaration forms, and security checks upon entry.

[0261] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0262] In the present invention, a specific embodiment for implementing a system centered on a user terminal, a server, an emotion engine, and data exchange between these will be described.

[0263] Advance preparation before departure and image recognition

[0264] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[0265] Device: The captured images are temporarily stored on the device, which then uploads them to the server.

[0266] Server: The server analyzes the received image data. It uses AI technology to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item. The result of the determination is sent to the user's device.

[0267] Travel information collation and notification

[0268] Server: After obtaining the image analysis results, the server checks the user's passport information. Specifically, it checks the user's belongings against the regulations of the destination country and the airline to see if they violate any rules. If any prohibited or restricted items are detected, the server notifies the user.

[0269] Terminal: The user terminal displays the notification received from the server and prompts the user to take necessary measures, such as changing prohibited items into checked baggage.

[0270] Auto-generation of customs declaration documents

[0271] Server: The server automatically generates the necessary customs declaration forms based on the list of recognized items, including determining whether the items require declaration.

[0272] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed to the user on the terminal, allowing the user to follow the instructions and prepare for a smooth customs process.

[0273] Smuggling detection and emotion recognition at immigration

[0274] Terminal: During immigration procedures, the user's terminal captures audio and video data and transmits it to the server in real time. The emotion engine also recognizes the user's emotions from the audio and video data.

[0275] Server: The server analyzes the received audio and video data and the emotion recognition results obtained from the emotion engine to detect suspicious behavior or speech. For example, it determines if the user makes contradictory statements or exhibits specific non-verbal behavior.

[0276] Server: When suspicious behavior or negative sentiment is detected, the server notifies security personnel in real time, allowing for any necessary further investigations or responses to be taken promptly.

[0277] Emotion-based personalization

[0278] Server: Based on the emotion recognition results obtained from the emotion engine, the server provides personalized notifications and advice to the user. For example, if tension or anxiety is detected, the server provides guidance and advice on how to relax.

[0279] Device: The user's device receives this personalized notification and provides the user with appropriate information to support a comfortable travel experience.

[0280] Specific examples

[0281] Example 1: Image recognition before departure

[0282] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[0283] Example 2: Automatic generation of customs declaration documents

[0284] When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. By presenting this form at the customs office, the user can smoothly proceed with customs clearance.

[0285] Example 3: Contraband detection and emotion recognition

[0286] If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If unnatural behavior is detected, the server immediately notifies security, and additional measures are taken. If the emotion engine detects tension or anxiety in the user, the server sends a notification offering advice to the user on how to relax.

[0287] The system will reduce immigration issues, significantly improve the efficiency of security and customs inspections, and provide appropriate support based on the user's emotional state, ensuring a comfortable travel experience.

[0288] The processing flow will be explained below.

[0289] Step 1:

[0290] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[0291] Step 2:

[0292] Device: Images are temporarily stored on the device. Once all items have been photographed, the image data will be ready to upload.

[0293] Step 3:

[0294] On the device: Upload the captured image to the server. Display a notification to the user when the upload is complete.

[0295] Step 4:

[0296] Server: Analyzes the received image data. Using AI technology, it recognizes and classifies the objects in each image.

[0297] Step 5:

[0298] Server: Compares the analysis results with the destination country's regulations and airline rules to identify prohibited or restricted items.

[0299] Step 6:

[0300] Server: If prohibited or restricted items are detected, it generates a list of them and creates notification data.

[0301] Step 7:

[0302] Terminal: Receives notification data sent from the server and displays it to the user. The user checks the displayed information and takes necessary measures.

[0303] Step 8:

[0304] Server: Based on the list of recognized items, a customs declaration form is automatically generated, listing the items that must be declared.

[0305] Step 9:

[0306] Server: Sends automatically generated customs declaration documents to the user's terminal.

[0307] Step 10:

[0308] Terminal: Displays the customs declaration documents sent to the user's terminal. The user checks the contents of the documents and prepares for customs procedures.

[0309] Step 11:

[0310] Terminal: During immigration procedures, audio and video data is captured and sent to the server in real time. An emotion engine also runs to recognize the user's emotions.

[0311] Step 12:

[0312] Server: Analyzes the received audio and video data and emotion recognition results obtained from the emotion engine to detect suspicious behavior or speech. For example, it determines if the user makes contradictory statements or engages in specific non-verbal behavior.

[0313] Step 13:

[0314] Server: When suspicious behavior or negative sentiment is detected, security personnel are notified in real time, allowing for any necessary further investigation or response to be taken promptly.

[0315] Step 14:

[0316] Server: Provides personalized notifications and advice to users based on the emotion recognition results obtained from the emotion engine. For example, if tension or anxiety is detected, it sends guidance and advice on how to relax.

[0317] Step 15:

[0318] Device: The user's device receives the personalized notification and displays appropriate information to the user, who can then check the notification and take action to relax.

[0319] Example 2

[0320] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] In recent years, with the increase in cross-border travel, safety and efficiency at immigration are required, but current systems have difficulty in detecting the bringing in of prohibited or restricted items in advance. Another issue is that customs declaration procedures and security checks at the point of entry require time and effort from users. This reduces convenience for travelers and the efficiency of security and customs inspections.

[0322] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for taking an image of the carried items using the user terminal, a means for temporarily saving the image taken by the terminal and uploading it to the server, a means for analyzing the image data received by the server and classifying the identified items, a means for determining whether the identified items fall under prohibited or restricted items, and a means for notifying the user terminal of the determination result. This enables the user to quickly check the carried items before departure and take necessary measures. In addition, customs declaration procedures are simplified and security checks at the time of entry are made more efficient.

[0323] A "user terminal" is an electronic device that a user uses to take pictures of their belongings and upload the data to a server.

[0324] The "means for taking an image" is a function that allows a user to capture an image of an item carried by the user using the user terminal.

[0325] "Terminal" refers to a user terminal or a device that has the function of temporarily storing captured images and uploading them to a server.

[0326] The "means for temporarily saving images" is a function for temporarily saving captured images on the terminal.

[0327] The "server" is a central management system that analyzes the received image data, classifies the identified items, and notifies the results to the user terminal.

[0328] The "means for analyzing image data" refers to algorithms or software used by the server to analyze the image data received.

[0329] The "means for classifying identified items" is a function by which the server identifies items from image data and classifies them into categories.

[0330] The "means of determining whether an item is a prohibited or restricted item" is a function that uses analyzed data to check whether the identified item violates the regulations of the destination or airline.

[0331] The "means for notifying the user terminal of the determination result" is a function by which the server sends a message to the user terminal to notify the user of the determination result.

[0332] "Travel information" is data such as the user's passport information and travel destination information.

[0333] The "means for automatically generating customs declaration documents" is a function that allows the server to automatically generate customs declaration documents based on the list of carried items.

[0334] The present invention provides a system for travelers to efficiently and safely complete immigration procedures. The system includes a user terminal, a server, and a network for exchanging data between them.

[0335] Hardware and software used

[0336] User device: A device such as a smartphone or tablet that takes pictures of personal items and uploads the data to a server. This device must have a camera and internet connection.

[0337] Server: The server is a central management system that receives and analyzes image data. Deep learning frameworks such as "TensorFlow" and "YOLO" are used for analysis. MySQL (registered trademark) and PostgreSQL are used as databases.

[0338] Network: Data exchange requires an internet connection and uses the HTTP or HTTPS protocol.

[0339] Specific system processing

[0340] Preparation before departure and image recognition

[0341] User: Before leaving the country, the user uses their smartphone to take pictures of their belongings. For example, they take pictures of the items in their suitcase one by one with the smartphone camera.

[0342] On the device: The captured image is temporarily stored in the device's storage, and then the device uploads the image to the server using the HTTP protocol.

[0343] Server: The server analyzes the received image data using TensorFlow and YOLO to automatically identify and classify the items. For example, it detects a "lighter" or "laptop" in the image. It compares the item with rules stored in a database to determine whether it is a prohibited or restricted item. The result of the determination is notified to the user's device.

[0344] Travel information collation and notification

[0345] Server: After receiving the image analysis results, the server compares the passport information with the regulations of the destination. For example, it can obtain the latest regulations of the destination through API and check whether the belongings violate the regulations. If any prohibited or restricted items are detected, the server will notify the user.

[0346] Device: The user device displays the notification received from the server, for example, a pop-up with specific instructions such as "Lighters are not allowed on board. Please put them in checked luggage."

[0347] Auto-generation of customs declaration documents

[0348] Server: The server automatically generates customs declaration forms based on the list of carried items. For example, it creates the declaration forms using a template engine such as "Jinja2." The generated forms are then sent to the user's device.

[0349] Terminal: The user terminal receives the automatically generated customs declaration form and displays it to the user, who then goes through customs procedures accordingly.

[0350] Smuggling detection and emotion recognition at immigration

[0351] Terminal: During immigration procedures, the user's terminal uses a camera and microphone to capture audio and video data, which is then transmitted to the server in real time. The emotion engine then recognizes the user's emotions from the audio and video data.

[0352] Server: The server performs analysis using emotion recognition engines such as Microsoft® Azure® Cognitive Services and IBM Watson®. For example, it detects the user's tension or anxiety. If suspicious behavior or negative emotions are detected, the server notifies security personnel in real time.

[0353] Emotion-based personalization

[0354] Server: The server provides personalized notifications and advice to users based on the emotion recognition results. For example, if tension is detected, the server sends a notification such as "Take a deep breath to relax."

[0355] Device: The user device receives this personalized notification and displays appropriate information to the user.

[0356] Examples of specific examples and prompts

[0357] Example 1: Before leaving the country, a user takes a picture of their baggage with their smartphone, and the device sends the picture to a server. If the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as checked baggage."

[0358] Example 2: If a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. The user then presents the form to the customs officer.

[0359] Example 3: If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If unnatural behavior is detected, the server immediately notifies security and additional measures are taken. If the emotion engine detects tension or anxiety in the user, the server sends a notification offering advice to the user on how to relax.

[0360] Example prompt sentence:

[0361] "Please take pictures of your belongings before you leave the country to make sure they do not contain any prohibited items."

[0362] "Auto-generate customs declaration documents that comply with the regulations of your destination."

[0363] "Detect smuggling and suspicious behavior in real time at the point of entry and provide security notifications based on emotion recognition results."

[0364] The system will reduce immigration issues, significantly improve the efficiency of security and customs procedures, and provide appropriate support based on the user's emotional state, ensuring a comfortable travel experience.

[0365] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0366] Step 1:

[0367] User: Before leaving the country, the user uses their smartphone to take a picture of their belongings.

[0368] Input: Image of belongings

[0369] Output: Image file saved in smartphone storage

[0370] Specific operation: The camera app on the smartphone is launched and each item is photographed individually. The photographed images are saved to the smartphone's internal storage or SD card.

[0371] Step 2:

[0372] On your device: Upload the saved image to the server.

[0373] Input: Image file saved in smartphone storage

[0374] Output: Image data sent to the server

[0375] What it does: It uses a smartphone app to send image data to a server via the HTTPS protocol, and displays a progress bar until all images taken by the user have been uploaded.

[0376] Step 3:

[0377] Server: Analyzes the received image data and identifies the item.

[0378] Input: Image data uploaded to the server

[0379] Output: List of identified items

[0380] How it works: The server runs deep learning models using TensorFlow and YOLO to identify the objects in each image, for example, determining whether the image contains a "lighter" or a "laptop."

[0381] Step 4:

[0382] Server: Determines whether the identified item is a prohibited or restricted item.

[0383] Input: List of identified items

[0384] Output: List of prohibited and restricted items

[0385] Specific operation: The system checks the regulations of the destination country stored in the database to determine whether the identified item is a prohibited or restricted item. For example, a "lighter" is detected as a prohibited item on board.

[0386] Step 5:

[0387] Server: Notifies the user of the result.

[0388] Input: List of prohibited and restricted items

[0389] Output: Notification message to the user's terminal

[0390] Specific operation: The server sends a push notification to the user's smartphone. For example, a message such as "Lighters are not permitted on board. Please check them in." is displayed.

[0391] Step 6:

[0392] Server: Automatically generate customs declaration documents.

[0393] Input: List of belongings

[0394] Output: Auto-generated customs declaration form

[0395] What happens: The server uses a template engine (e.g., Jinja2) to generate a customs declaration form from the list of items carried. The documents include details of the items that need to be declared.

[0396] Step 7:

[0397] Server: Sends automatically generated customs declaration documents to the user's terminal.

[0398] Input: Auto-generated customs declaration form

[0399] Output: Customs declaration document displayed on user's terminal

[0400] Specific operation: The server generates a customs declaration form, which is sent to the user's smartphone and displayed on the device. The user can save it as a PDF or present it directly to customs.

[0401] Step 8:

[0402] Device: During immigration procedures, a camera and microphone are used to capture audio and video data and transmit it to a server.

[0403] Input: Audio and video data

[0404] Output: Audio and video data sent to the server

[0405] Specific operation: During immigration procedures, the smartphone's camera and microphone are activated, and real-time audio and video recordings are sent to the server via the HTTPS protocol.

[0406] Step 9:

[0407] Server: Analyzes audio and video data and recognizes emotions.

[0408] Input: Audio and video data sent to the server

[0409] Output: Emotion recognition result

[0410] Specific operation: The server uses Microsoft Azure Cognitive Services and IBM Watson to analyze emotions from voice and recognize facial expressions from video to determine the user's emotional state.

[0411] Step 10:

[0412] Server: Notifies security personnel if suspicious behavior or negative sentiment is detected.

[0413] Input: Emotion recognition results

[0414] Output: Notification alert to security personnel

[0415] What it does: When abnormal behavior or negative sentiment is detected, the server sends real-time notifications to security personnel using a security alert system (e.g., PagerDuty).

[0416] Step 11:

[0417] Server: Provides personalized notifications and advice to users based on emotion recognition results.

[0418] Input: Emotion recognition results

[0419] Output: A personalized notification message

[0420] Specific operation: When a user feels nervous or anxious, the server generates an appropriate message and sends it to the user's smartphone, including advice such as "Take a deep breath to relax."

[0421] (Application example 2)

[0422] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0423] Immigration procedures, such as baggage inspections and customs procedures, are cumbersome and time-consuming, making it easy for fraud and violations to be overlooked. Furthermore, there is an insufficient system for recognizing suspicious passenger behavior and emotional states in real time and responding immediately, increasing security risks. These issues increase the burden on both travelers and inspectors, and create challenges for the overall travel experience.

[0424] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0425] In this invention, the server includes means for taking images of carried items using a user terminal, means for analyzing the taken images by the server and detecting prohibited or restricted items, means for notifying the user of the detection results, means for capturing audio and video data in real time and recognizing emotions using an emotion engine, and means for analyzing the recognition results of the emotion engine by the server to detect suspicious actions or behavior and notify security personnel in real time. This enables immigration procedures to be carried out quickly and accurately, and enables early detection and response to violations and fraudulent acts.

[0426] A "user terminal" is a device for taking pictures of personal items and capturing audio and video data.

[0427] A "server" is a central system that receives, analyzes, and processes data sent from user terminals.

[0428] "Personal effects" refers to all items that a user carries with them when entering or leaving a country.

[0429] "Means for taking images" refers to the function of capturing images of carried items using the camera of the user terminal.

[0430] "Prohibited or restricted items" are items that are prohibited or restricted from being carried on board by the airline or the country of destination.

[0431] "Means for detection" refers to the function of analyzing the image data received by the server and identifying prohibited or restricted items.

[0432] "Means for notifying" refers to a function that sends the detection results to the user terminal and prompts the user to take appropriate action.

[0433] "Means for capturing audio and video data in real time" refers to the ability to record environmental sounds and videos in real time using the camera and microphone of the user terminal.

[0434] "Emotion engine" refers to an analysis function for recognizing the user's emotional state from audio and video data.

[0435] "Suspicious behavior and speech" refers to abnormal behavior or speech that is not in line with normal patterns of behavior and is deemed to pose a security risk.

[0436] "Security Officer" refers to the personnel who conduct inspections, monitoring, and respond as necessary.

[0437] "Means for analyzing the recognition results of the emotion engine by the server" refers to a function for the server to perform behavioral analysis based on data obtained from the emotion engine.

[0438] "Means of notifying security personnel in real time" refers to the function of immediately sending an alert to security personnel when abnormal behavior or speech is detected.

[0439] The present invention is a system centered around a user terminal, a server, an emotion engine, and the exchange of data between them. The main components of this system and their functions are specifically shown below.

[0440] Hardware and software used

[0441] User device: Smartphone or dedicated device

[0442] Server: Cloud Server

[0443] Emotion Engine: EmotionEngine (emotion recognition module)

[0444] Image processing software: OpenCV

[0445] Server communication library: requests

[0446] Customs declaration generation software: TaxDeclarationGenerator (automatic generation module)

[0447] Data processing and calculation

[0448] 1. Taking and analyzing images of personal belongings

[0449] When a user arrives at the airport, they use the camera on their device to take a picture of their belongings. The captured image is temporarily saved on the device. The image is then encoded using OpenCV, an image processing software, and uploaded to the server. The server receives the uploaded image data and uses AI technology to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item and notifies the user's device of the result.

[0450] 2. Travel information collation and notification

[0451] The server compares the user's passport information with the regulations of the destination country based on the image analysis results. If a prohibited or restricted item is detected, the server notifies the user of the results and prompts the user to take necessary measures. For example, if a prohibited item such as a "lighter" is detected, the server notifies the user that "lighters are not permitted on board. Please check them in as luggage."

[0452] 3. Automatic generation of customs declaration documents

[0453] The server automatically generates a customs declaration form based on the list of recognized items. This process includes determining whether the items are subject to declaration. The automatically generated customs declaration form is sent to the user's terminal, where the user can use it for customs procedures.

[0454] 4. Emotion Recognition and Suspicious Behavior Detection at Entry

[0455] During immigration inspections, the user's device captures audio and video data in real time and sends it to the server. The emotion engine recognizes the user's emotions from this data, and the server analyzes the recognition results. If suspicious behavior or speech is detected, the server immediately notifies security personnel. For example, if the user's history is normal but significant tension or anxiety is detected, the server will transmit that information to security personnel in real time.

[0456] 5. Personalized notifications and relaxation guide

[0457] Based on the emotion recognition results, the server provides personalized notifications and advice to the user. For example, if tension or anxiety is detected, a notification offering guidance and voice advice on how to relax will be sent to the user's device. This reduces the user's stress and provides a comfortable travel experience.

[0458] Examples of concrete examples and prompts

[0459] Example: Security Assistant App Usage Scenario

[0460] When a user arrives at the airport, they take a photo of their belongings and upload it before checking in. The image is analyzed and a notification is sent to them that the detected "lighter" is not permitted. Later, if the user becomes nervous during immigration, the emotion engine detects this and the server notifies the security officer.

[0461] Prompt Sentence Examples

[0462] Consider a scenario where a user takes and uploads a picture of their baggage at an airport. The image is analyzed to classify the items according to airline and destination country regulations, detect any illegal items, and then recognize the user's emotions in real time, notifying security if suspicious behavior is detected.

[0463] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0464] Step 1:

[0465] The user arrives at the airport and takes a picture of their belongings using the camera on the user terminal.

[0466] Input: Image of belongings

[0467] Output: Captured image file

[0468] Specific operation: Capture an image of the item with the user's device's camera app and save it as an image file on the device.

[0469] Step 2:

[0470] The captured image files are encoded using OpenCV and uploaded to the server.

[0471] Input: Image file

[0472] Output: Send image data to the server

[0473] Specific operation: Encodes an image using the OpenCV library on the user device, and sends the image data to the server as an HTTP POST request using the requests library.

[0474] Step 3:

[0475] The server analyzes the received image data and uses AI technology to recognize and classify the items.

[0476] Input: Encoded image data

[0477] Output: Analysis results (list of recognized items, detection results of prohibited and restricted items)

[0478] How it works: The server uses an AI model to analyze image data, automatically classify recognized items, and generate detection results for prohibited and restricted items.

[0479] Step 4:

[0480] The server notifies the user of the analysis results and prompts the user to take necessary measures.

[0481] Input: Analysis results (detection results of prohibited and restricted items)

[0482] Output: Notification message to the user's terminal

[0483] Specific operation: The server generates a notification message based on the detection result and sends it to the user's device. For example, if a prohibited item such as a "lighter" is detected, the server notifies the user by saying, "Lighters are prohibited from being carried on board. Please put them in checked baggage."

[0484] Step 5:

[0485] The server compares the analysis results with travel information to check whether the items being carried are in compliance with regulations.

[0486] Input: Analysis results, travel information (passport information)

[0487] Output: Whether or not there is a violation and a notification message

[0488] Specific operation: The server compares the travel information with the analysis results, and if any illegal items are found, sends a message to the user's device.

[0489] Step 6:

[0490] During immigration control, the user terminal captures audio and video data in real time and transmits it to the server.

[0491] Input: Real-time audio and video data

[0492] Output: Sending audio and video data to the server

[0493] Specific operation: Captures environmental sounds and videos in real time using the camera and microphone of the user device and sends them to the server.

[0494] Step 7:

[0495] The server analyzes the received audio and video data using an emotion engine to recognize the user's emotions.

[0496] Input: Real-time audio and video data

[0497] Output: Emotion recognition result

[0498] Specific operation: The server uses EmotionEngine to analyze audio and video data and recognize the user's emotional state.

[0499] Step 8:

[0500] The server analyzes the emotion recognition results and notifies security personnel if any suspicious behavior or speech is detected.

[0501] Input: Emotion recognition results

[0502] Output: A message to notify the security officer

[0503] Specific operation: The server analyzes the emotion recognition results, and if any suspicious behavior or speech is detected, it sends a message to the security officer in real time.

[0504] Step 9:

[0505] The server provides personalized notifications and advice to the user based on the emotion recognition results.

[0506] Input: Emotion recognition results

[0507] Output: A personalized notification message to the user

[0508] Specific operation: If the server detects tension or anxiety based on the emotion recognition results, it sends a notification message to the user's device containing relaxation guidance and advice.

[0509] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0510] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0511] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0512] [Second embodiment]

[0513] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0514] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0515] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0516] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0517] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0518] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0519] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0520] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0521] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0522] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0523] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0524] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0525] The embodiment of the present invention will be described with a focus on a user terminal, a server, and data exchange between them.

[0526] Advance preparation before departure and image recognition

[0527] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[0528] Device: The captured images are temporarily stored on the device, which then uploads them to the server.

[0529] Server: The server analyzes the received image data. During the analysis, AI technology is used to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item. The result of the determination is sent to the user's device.

[0530] Travel information collation and notification

[0531] Server: After obtaining the image analysis results, the server checks the user's passport information. Specifically, it compares the user's belongings with the destination country's regulations and airline regulations to see if they violate any rules. If prohibited or restricted items are detected, the server notifies the user.

[0532] Terminal: The user terminal displays the notification received from the server and prompts the user to take necessary measures, such as changing prohibited items into checked baggage.

[0533] Auto-generation of customs declaration documents

[0534] Server: The server automatically generates the necessary customs declaration forms based on the list of recognized items, including determining whether the items require declaration.

[0535] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed to the user on the terminal, allowing the user to follow the instructions and prepare for a smooth customs process.

[0536] Smuggling detection at the point of entry

[0537] Device: During immigration procedures, the user's device captures audio and video data and transmits it to a server in real time, allowing the user's actions and behavior during the procedure to be monitored.

[0538] Server: The server analyzes the received audio and video data to detect suspicious behavior or actions, such as when a user makes contradictory statements or exhibits specific non-verbal behavior.

[0539] Server: When suspicious activity is detected, the server notifies security personnel in real time, allowing for any necessary further investigations or responses to be taken promptly.

[0540] Specific examples

[0541] Example 1: Image recognition before departure

[0542] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[0543] Example 2: Automatic generation of customs declaration documents

[0544] When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. By presenting this form at the customs office, the user can smoothly proceed with customs clearance.

[0545] Example 3: Detecting contraband

[0546] If a user behaves suspiciously during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security, and additional measures are taken.

[0547] This system will reduce immigration issues and significantly improve the efficiency of security and customs inspections.

[0548] The processing flow will be explained below.

[0549] Step 1:

[0550] User: Takes pictures of their belongings using a smartphone or dedicated device. The user takes pictures of all items in turn and records the necessary information.

[0551] Step 2:

[0552] Device: Temporarily stores captured images. Once all items have been photographed, the image data will be ready to upload.

[0553] Step 3:

[0554] On the device: Upload the captured image to the server. Display a notification to the user when the upload is complete.

[0555] Step 4:

[0556] Server: Analyzes the received image data. Using AI technology, it recognizes and classifies the objects in each image.

[0557] Step 5:

[0558] Server: Compares the analysis results with the destination country's regulations and airline rules to identify prohibited or restricted items.

[0559] Step 6:

[0560] Server: If prohibited or restricted items are detected, it generates a list of them and creates notification data.

[0561] Step 7:

[0562] Terminal: Receives notification data sent from the server and displays it to the user. The user checks the displayed information and takes necessary measures.

[0563] Step 8:

[0564] Server: Based on the list of recognized items, a customs declaration form is automatically generated, listing the items that must be declared.

[0565] Step 9:

[0566] Server: Sends automatically generated customs declaration documents to the user's terminal.

[0567] Step 10:

[0568] Terminal: Displays the customs declaration documents sent to the user's terminal. The user checks the contents of the documents and prepares for customs procedures.

[0569] Step 11:

[0570] Terminal: During immigration procedures, audio and video data is captured and transmitted to a server in real time.

[0571] Step 12:

[0572] Server: Analyzes received audio and video data to detect suspicious behavior and speech, including non-verbal behavior and contradictory statements.

[0573] Step 13:

[0574] Server: If suspicious activity is detected, it notifies security personnel in real time and sends the analysis results as additional information.

[0575] Step 14:

[0576] Security personnel: Receive notifications from the server and promptly carry out any necessary additional checks or actions.

[0577] Example 1

[0578] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0579] Conventional immigration control systems require users to perform numerous manual tasks when handling carry-on baggage and customs declaration procedures, resulting in inefficiency and a high likelihood of errors. Furthermore, detecting smuggling during immigration inspections relies on human resources, making thorough monitoring difficult. Therefore, there is a need for a system that can improve efficiency and prevent fraud in each process of carry-on baggage management, customs declaration, and immigration inspection.

[0580] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0581] In this invention, the server includes means for taking images of carried items using a user terminal, means for temporarily saving the images taken by the terminal, means for uploading the images saved by the terminal to the server, means for analyzing the image data received by the server using AI technology to recognize and classify the items, means for determining whether the items fall under prohibited or restricted items, and means for transmitting the determination result to the user terminal. This automates the management of carried items, reduces the burden on users, and enables the prevention of fraudulent activities and the efficiency of immigration procedures.

[0582] A "user terminal" is an electronic device such as a mobile information terminal or a smartphone that is intended to be operated by a user.

[0583] "Personal effects" refers to all items that a user carries with them when traveling or moving.

[0584] The "means for taking an image" is a function for taking an image of a carried item using a camera function provided in the user terminal.

[0585] The "temporary storage means" is a function for storing captured images in a temporary storage device within the user terminal.

[0586] A "server" is a remote data processing device that receives and processes data sent from a user terminal.

[0587] "Uploading means" is a function for transmitting data from a user terminal to a server.

[0588] "Means for analyzing image data using AI technology" refers to a processing function that uses AI technology on a server to identify and classify objects contained in images.

[0589] "Means for recognizing and classifying items" refers to a function that identifies and classifies the name and type of an item from image data analyzed by AI.

[0590] "Means for determining whether a recognized item is a prohibited or restricted item" is a function that determines whether the recognized item violates the laws and regulations of the destination country.

[0591] The "means for transmitting the determination result to the user terminal and notifying the user" is a function for transmitting the determination result to the user terminal and displaying a notification to the user.

[0592] "Travel information" refers to all information related to a user's travel, such as passport information and airline ticket information.

[0593] "Means of matching" refers to the function of comparing the image analysis results with travel information to check for matches or violations.

[0594] The "means for automatically generating customs declaration documents" is a function that automatically creates the necessary customs declaration documents based on the analyzed carry-on item data.

[0595] The "means for displaying the automatically generated customs declaration document on the user terminal" is a function for transmitting the generated customs declaration document to the user terminal so that the user can view it on the terminal.

[0596] The present invention relates to a system for improving the efficiency of carry-on baggage management and customs declaration procedures at the time of immigration and for strengthening smuggling detection. Specific embodiments of the system are described below.

[0597] Preparation before departure and image recognition

[0598] User: Before leaving the country, the user takes pictures of their belongings using their smartphone or a dedicated device. This is done using the smartphone's camera function. For example, the user can take pictures of each item they put in their suitcase, obtaining image data of all of their belongings.

[0599] On the device: The captured images are saved in a temporary folder (local storage) on the user's device. They are then automatically uploaded from the device to the server. This upload process is carried out via a dedicated application on the device.

[0600] Server: The server analyzes the received image data using AI technology. Specifically, it uses image analysis libraries such as TensorFlow and OpenCV to recognize and classify objects. For example, objects in the image are recognized as "lighter," "bottle," "book," etc.

[0601] Server: Based on the analysis results, the server determines whether the item is a prohibited or restricted item. This involves checking it against a list of prohibited items stored on the server. The result of the determination is then sent to the user's device. The user is notified and can take the necessary measures accordingly.

[0602] Travel information collation and notification

[0603] Server: After obtaining the image analysis results, the server checks the user's passport information (electronic passport data) and travel information to ensure that the belongings comply with the regulations of the destination country and airline.

[0604] Server: If a prohibited or restricted item is detected, the server sends that information to the user's device and notifies the user of the appropriate action to take, such as "Lighters are not permitted on board. Please put them in checked baggage."

[0605] Auto-generation of customs declaration documents

[0606] Server: The server automatically generates the necessary customs forms based on the list of recognized items, including determining whether the items require declaration.

[0607] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed in a PDF viewer or dedicated application, allowing the user to follow the instructions and prepare for smooth customs procedures.

[0608] Smuggling detection at the point of entry

[0609] Terminal: During immigration control, the user's terminal captures audio and video data and transmits it to the server in real time using the terminal's built-in camera and microphone.

[0610] Server: The server analyzes the received audio and video data to detect suspicious behavior and speech. For example, it uses voice recognition and facial recognition systems to analyze the user's speech and non-verbal behavior (e.g., facial expressions and body movements).

[0611] Server: If any suspicious activity is detected, the server immediately notifies security personnel and prompts them to take any necessary further investigations or actions, thereby monitoring potential smuggling activity in real time.

[0612] Specific examples

[0613] Example 1: Before leaving the country, a user takes a picture of their baggage with their smartphone, and the device sends the picture to a server. If the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as checked baggage."

[0614] Example 2: When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. The user then presents the form at the customs office and the customs clearance process goes smoothly.

[0615] Example 3: If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security, and additional measures are taken.

[0616] This system will reduce immigration issues and significantly improve the efficiency of security and customs inspections.

[0617] Prompt Sentence Examples

[0618] Example: "How can I take a picture of my baggage before leaving the country and upload it to a server, so that AI can recognize and classify the items and notify the user if any prohibited or restricted items are detected?"

[0619] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0620] Step 1:

[0621] User: The user uses a smartphone or dedicated device to take pictures of each item in their baggage. Specifically, the user launches a camera app and takes pictures of each item of baggage.

[0622] Input: The physical state of the baggage.

[0623] Output: Digital image data.

[0624] Step 2:

[0625] Device: The captured image is automatically saved to a temporary folder on the user's device.

[0626] Input: Images taken with a camera app on a smartphone or dedicated device.

[0627] Output: Image file saved in the device's local storage.

[0628] Step 3:

[0629] On the device: To upload the saved image to the server, send the image data via the API. Specifically, send the image data to the server using an HTTP request.

[0630] Input: An image file stored in local storage.

[0631] Output: Image data sent to the server.

[0632] Step 4:

[0633] Server: The server analyzes the received image data using AI technology, for example, using image analysis libraries such as TensorFlow or OpenCV, and applies a model to recognize objects in the image.

[0634] Input: Image data sent to the server.

[0635] Output: Identified item information as a result of analysis (e.g. lighter, plastic bottle, book, etc.).

[0636] Step 5:

[0637] Server: Based on the analysis results, it determines whether the item is a prohibited or restricted item, and in this process, it checks against the prohibited items list.

[0638] Input: Identified item information as a result of analysis.

[0639] Output: Judgment result (e.g. prohibited item, restricted item, permitted item).

[0640] Step 6:

[0641] Server: Sends the judgment result to the user device. Specifically, it generates JSON data in the form of a notification and sends it to the user device.

[0642] Input: The result of the test.

[0643] Output: Notification data sent to the user device.

[0644] Step 7:

[0645] Terminal: The user terminal analyzes the received notification data and displays it to the user. Specifically, it uses a pop-up notification or an alert dialog to alert the user.

[0646] Input: Notification data sent by the server.

[0647] Output: The notification displayed on the user's screen.

[0648] Step 8:

[0649] Server: Compares the user's passport and travel information with the image analysis results, specifically by accessing a database and comparing them with the destination country's regulations and airline regulations.

[0650] Input: Passport information, travel information, and image analysis results.

[0651] Output: Verification data as a result of the match (e.g., whether or not there is a violation).

[0652] Step 9:

[0653] Server: If prohibited or restricted items are detected based on the matching results, the server notifies the user of this information.

[0654] Input: Verification data as a match result.

[0655] Output: The alert data sent to the user terminal.

[0656] Step 10:

[0657] Terminal: The user's terminal analyzes the received warning data and notifies the user of countermeasures. For example, it displays specific instructions such as "Lighters are prohibited from being carried on board. Please put them in checked baggage."

[0658] Input: The alert data sent by the server.

[0659] Output: The instructions displayed on the user's screen.

[0660] Step 11:

[0661] Server: Automatically generates customs declaration documents based on the list of recognized carry-on items. Specifically, it automatically fills in the necessary information in a template and generates the documents in PDF format.

[0662] Input: A list of recognized items.

[0663] Output: Auto-generated customs declaration form (PDF).

[0664] Step 12:

[0665] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed for the user to view.

[0666] Input: Auto-generated customs declaration form (PDF).

[0667] Output: Customs declaration document displayed on user's terminal.

[0668] Step 13:

[0669] Terminal: During immigration control, the user terminal captures audio and video data and transmits it to the server in real time using the built-in camera and microphone.

[0670] Input: Audio and video data from immigration inspection.

[0671] Output: Audio and video data sent to the server.

[0672] Step 14:

[0673] Server: The server analyzes the received audio and video data and detects suspicious behavior or speech. Specifically, it uses voice recognition and facial recognition systems.

[0674] Input: Audio and video data sent to the server.

[0675] Output: Evaluation data of behavior and speech as the analysis result.

[0676] Step 15:

[0677] Server: If suspicious activity is detected, the server notifies security personnel in real time, facilitating further investigation and response.

[0678] Input: Evaluation data of behavior and speech as the analysis results.

[0679] Output: Alert data sent to security personnel.

[0680] This processing step reduces immigration issues and significantly improves the efficiency of security and customs inspections.

[0681] (Application example 1)

[0682] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0683] When traveling internationally, it is cumbersome to check whether your belongings comply with regulations before leaving the country, and it takes time to fill out customs declaration forms. Furthermore, there is a lack of means to quickly identify travelers exhibiting suspicious behavior during immigration inspections, making it difficult to ensure security. There is a need for a solution to these problems, making immigration procedures more efficient and improving security.

[0684] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0685] In this invention, the server includes means for taking images of carried items using a user terminal, means for analyzing the images taken by the server to detect prohibited or restricted items, means for notifying the user of the detection results, means for automatically generating customs declaration forms and displaying them on the user terminal, means for capturing audio and video data during immigration inspection and transmitting it to the server in real time, means for analyzing the audio and video data in the server to detect suspicious behavior, and means for notifying security personnel when suspicious behavior is detected. This enables quick confirmation of carried items before departure, automates the creation of customs declaration forms, and strengthens security inspections at the time of entry.

[0686] A "user terminal" is an electronic device that is carried and operated by a user, and is a device for taking images of carried items and receiving notifications.

[0687] A "server" is a computer system connected to a network that stores, analyzes, and processes data.

[0688] "Personal effects" are items that a user carries with them when traveling or going out.

[0689] The "means for taking an image" is a mechanism for taking a photo of a carried item using the camera function installed in the user terminal.

[0690] The "means for analyzing images" refers to algorithms or software for processing the image data received by the server and detecting specific items.

[0691] "Prohibited items" are items that are prohibited from being possessed or brought into the country under laws and regulations.

[0692] "Restricted items" are items whose possession or import is restricted based on permits or conditions.

[0693] "Means for notifying" refers to a technology for transmitting information from a server to a user terminal in real time and informing the user.

[0694] "Customs Declaration Form" means an official document used to declare possession of goods to customs.

[0695] "Automatic generation means" refers to a system that uses a server algorithm to automatically generate the necessary documents.

[0696] "Means for capturing audio and video data" refers to the process of acquiring audio and video using a microphone and camera installed on the user terminal.

[0697] "Means for transmitting in real time" refers to Internet communication technology for instantly transmitting data to a server.

[0698] "Means for detecting suspicious behavior" refers to technology that automatically detects suspicious behavior and speech based on audio and video data on the server.

[0699] "Means for notifying security personnel" refers to a system for sending warning information to security personnel when the server detects suspicious activity.

[0700] The present invention provides a system for improving the efficiency of baggage management before departure and security checks at the time of entry through data exchange between a user terminal and a server. Specific embodiments of the system are described below.

[0701] Baggage management before departure

[0702] 1. Taking pictures using a user device

[0703] A user takes a picture of their belongings with a user terminal such as a smartphone, and image data of the baggage is acquired using the camera of the smartphone.

[0704] 2. Sending image data

[0705] The captured image data is uploaded from the user's device to a server via an internet connection, and the data is sent in real time.

[0706] 3. Server Image Analysis

[0707] The server analyzes the received image data using AI technology. Specifically, it uses an object recognition algorithm to identify items in the image and detect prohibited or restricted items. For this, image analysis software such as OpenCV or TensorFlow can be used.

[0708] 4. Notification of test results

[0709] The analysis results are sent to the user's device, where the user can check whether or not there are any prohibited or restricted items. If the user accidentally carries a prohibited item, the device will notify the user in real time.

[0710] 5. Automatic generation of customs declaration documents

[0711] The server automatically generates customs declaration forms based on the analyzed list of carried items, and the generated forms are sent to the user's terminal, where the user can view and print them.

[0712] Security checks at entry

[0713] 1. Audio and Video Capture

[0714] During immigration inspection, the user's device (smartphone) captures audio and video data in real time and transmits it to a server using the smartphone's microphone and camera.

[0715] 2. Real-time data analysis

[0716] The server analyzes the received audio and video data, using natural language processing technology for audio analysis and object detection and motion analysis technology for video analysis, to detect suspicious behavior or contradictory statements in real time.

[0717] 3. Security Notice

[0718] If any suspicious activity is detected, the server immediately notifies security personnel, allowing any necessary additional testing or response to be carried out promptly.

[0719] Examples of specific examples and prompts

[0720] Example 1: Image recognition before departure

[0721] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[0722] Example 2: Security checks at entry

[0723] If a user behaves suspiciously during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security personnel, who then take additional action.

[0724] Prompt for the generative AI model (luggage analysis example):

[0725] "Analyze the image to determine if it contains any illegal items. The image data is contained in the next argument, image_data."

[0726] The above embodiment is expected to improve the efficiency of immigration procedures and security.

[0727] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0728] Step 1:

[0729] A user takes a photo of their belongings with a user terminal such as a smartphone. The captured image data is acquired using the smartphone's camera. The input is the image data captured by the user's operation, and the output is the image data temporarily saved on the terminal.

[0730] Step 2:

[0731] The device uploads the captured image data to a server. The data is transmitted in real time via an internet connection. The input is the captured image data, and the output is the image data transmitted to the server.

[0732] Step 3:

[0733] The server analyzes the received image data using AI technology. Specifically, it uses object recognition algorithms (e.g., OpenCV or TensorFlow) to identify items in the image and detect prohibited or restricted items. The input is the image data sent to the server, and the output is the analysis results.

[0734] Step 4:

[0735] The server sends a notification to the user's device based on the analysis results. The notification includes details of the prohibited or restricted items detected. The input is the result data from the AI ​​analysis, and the output is a notification message sent to the user's device.

[0736] Step 5:

[0737] The server automatically generates customs declaration forms based on the analyzed list of carried items, including determining which items should be declared. The input is the list of items based on the analysis results, and the output is the generated customs declaration form.

[0738] Step 6:

[0739] The server sends the automatically generated customs declaration form to the user's terminal, where the user can check and print it. The input is the generated customs declaration form, and the output is the declaration form sent to the user's terminal.

[0740] Step 7:

[0741] During immigration control, the user's device (smartphone) captures audio and video data and transmits it to the server in real time. The data is acquired using the smartphone's microphone and camera. The input is the audio and video data captured at the immigration control site, and the output is the audio and video data transmitted to the server.

[0742] Step 8:

[0743] The server analyzes the received audio and video data. It uses natural language processing technology for audio analysis and object detection and motion analysis technology for video analysis. The input is the captured audio and video data, and the output is behavioral judgment data as the analysis result.

[0744] Step 9:

[0745] Based on the analysis results, the server determines whether there is any suspicious behavior or contradictory statements, and notifies security personnel if any suspicious behavior is detected. The input is behavior determination data, and the output is a notification message to the security personnel.

[0746] These steps ensure efficient and effective baggage checks before departure, automatic generation of customs declaration forms, and security checks upon entry.

[0747] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0748] In the present invention, a specific embodiment for implementing a system centered on a user terminal, a server, an emotion engine, and data exchange between these will be described.

[0749] Advance preparation before departure and image recognition

[0750] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[0751] Device: The captured images are temporarily stored on the device, which then uploads them to the server.

[0752] Server: The server analyzes the received image data. It uses AI technology to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item. The result of the determination is sent to the user's device.

[0753] Travel information collation and notification

[0754] Server: After obtaining the image analysis results, the server checks the user's passport information. Specifically, it checks the user's belongings against the regulations of the destination country and the airline to see if they violate any rules. If any prohibited or restricted items are detected, the server notifies the user.

[0755] Terminal: The user terminal displays the notification received from the server and prompts the user to take necessary measures, such as changing prohibited items into checked baggage.

[0756] Auto-generation of customs declaration documents

[0757] Server: The server automatically generates the necessary customs declaration forms based on the list of recognized items, including determining whether the items require declaration.

[0758] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed to the user on the terminal, allowing the user to follow the instructions and prepare for a smooth customs process.

[0759] Smuggling detection and emotion recognition at immigration

[0760] Terminal: During immigration procedures, the user's terminal captures audio and video data and transmits it to the server in real time. The emotion engine also recognizes the user's emotions from the audio and video data.

[0761] Server: The server analyzes the received audio and video data and the emotion recognition results obtained from the emotion engine to detect suspicious behavior or speech. For example, it determines if the user makes contradictory statements or exhibits specific non-verbal behavior.

[0762] Server: When suspicious behavior or negative sentiment is detected, the server notifies security personnel in real time, allowing for any necessary further investigations or responses to be taken promptly.

[0763] Emotion-based personalization

[0764] Server: Based on the emotion recognition results obtained from the emotion engine, the server provides personalized notifications and advice to the user. For example, if tension or anxiety is detected, the server provides guidance and advice on how to relax.

[0765] Device: The user's device receives this personalized notification and provides the user with appropriate information to support a comfortable travel experience.

[0766] Specific examples

[0767] Example 1: Image recognition before departure

[0768] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[0769] Example 2: Automatic generation of customs declaration documents

[0770] When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. By presenting this form at the customs office, the user can smoothly proceed with customs clearance.

[0771] Example 3: Contraband detection and emotion recognition

[0772] If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If unnatural behavior is detected, the server immediately notifies security, and additional measures are taken. If the emotion engine detects tension or anxiety in the user, the server sends a notification offering advice to the user on how to relax.

[0773] The system will reduce immigration issues, significantly improve the efficiency of security and customs inspections, and provide appropriate support based on the user's emotional state, ensuring a comfortable travel experience.

[0774] The processing flow will be explained below.

[0775] Step 1:

[0776] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[0777] Step 2:

[0778] Device: Images are temporarily stored on the device. Once all items have been photographed, the image data will be ready to upload.

[0779] Step 3:

[0780] On the device: Upload the captured image to the server. Display a notification to the user when the upload is complete.

[0781] Step 4:

[0782] Server: Analyzes the received image data. Using AI technology, it recognizes and classifies the objects in each image.

[0783] Step 5:

[0784] Server: Compares the analysis results with the destination country's regulations and airline rules to identify prohibited or restricted items.

[0785] Step 6:

[0786] Server: If prohibited or restricted items are detected, it generates a list of them and creates notification data.

[0787] Step 7:

[0788] Terminal: Receives notification data sent from the server and displays it to the user. The user checks the displayed information and takes necessary measures.

[0789] Step 8:

[0790] Server: Based on the list of recognized items, a customs declaration form is automatically generated, listing the items that must be declared.

[0791] Step 9:

[0792] Server: Sends automatically generated customs declaration documents to the user's terminal.

[0793] Step 10:

[0794] Terminal: Displays the customs declaration documents sent to the user's terminal. The user checks the contents of the documents and prepares for customs procedures.

[0795] Step 11:

[0796] Terminal: During immigration procedures, audio and video data is captured and sent to the server in real time. An emotion engine also runs to recognize the user's emotions.

[0797] Step 12:

[0798] Server: Analyzes the received audio and video data and emotion recognition results obtained from the emotion engine to detect suspicious behavior or speech. For example, it determines if the user makes contradictory statements or engages in specific non-verbal behavior.

[0799] Step 13:

[0800] Server: When suspicious behavior or negative sentiment is detected, security personnel are notified in real time, allowing for any necessary further investigation or response to be taken promptly.

[0801] Step 14:

[0802] Server: Provides personalized notifications and advice to users based on the emotion recognition results obtained from the emotion engine. For example, if tension or anxiety is detected, it sends guidance and advice on how to relax.

[0803] Step 15:

[0804] Device: The user's device receives the personalized notification and displays appropriate information to the user, who can then check the notification and take action to relax.

[0805] Example 2

[0806] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0807] In recent years, with the increase in cross-border travel, safety and efficiency at immigration are required, but current systems have difficulty in detecting the bringing in of prohibited or restricted items in advance. Another issue is that customs declaration procedures and security checks at the point of entry require time and effort from users. This reduces convenience for travelers and the efficiency of security and customs inspections.

[0808] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for taking an image of the carried items using the user terminal, a means for temporarily saving the image taken by the terminal and uploading it to the server, a means for analyzing the image data received by the server and classifying the identified items, a means for determining whether the identified items fall under prohibited or restricted items, and a means for notifying the user terminal of the determination result. This enables the user to quickly check the carried items before departure and take necessary measures. In addition, customs declaration procedures are simplified and security checks at the time of entry are made more efficient.

[0809] A "user terminal" is an electronic device that a user uses to take pictures of their belongings and upload the data to a server.

[0810] The "means for taking an image" is a function that allows a user to capture an image of an item carried by the user using the user terminal.

[0811] "Terminal" refers to a user terminal or a device that has the function of temporarily storing captured images and uploading them to a server.

[0812] The "means for temporarily saving images" is a function for temporarily saving captured images on the terminal.

[0813] The "server" is a central management system that analyzes the received image data, classifies the identified items, and notifies the results to the user terminal.

[0814] The "means for analyzing image data" refers to algorithms or software used by the server to analyze the image data received.

[0815] The "means for classifying identified items" is a function by which the server identifies items from image data and classifies them into categories.

[0816] The "means of determining whether an item is a prohibited or restricted item" is a function that uses analyzed data to check whether the identified item violates the regulations of the destination or airline.

[0817] The "means for notifying the user terminal of the determination result" is a function by which the server sends a message to the user terminal to notify the user of the determination result.

[0818] "Travel information" is data such as the user's passport information and travel destination information.

[0819] The "means for automatically generating customs declaration documents" is a function that allows the server to automatically generate customs declaration documents based on the list of carried items.

[0820] The present invention provides a system for travelers to efficiently and safely complete immigration procedures. The system includes a user terminal, a server, and a network for exchanging data between them.

[0821] Hardware and software used

[0822] User device: A device such as a smartphone or tablet that takes pictures of personal items and uploads the data to a server. This device must have a camera and internet connection.

[0823] Server: The server is a central management system that receives and analyzes image data. Deep learning frameworks such as "TensorFlow" and "YOLO" are used for analysis. MySQL and PostgreSQL are used as databases.

[0824] Network: Data exchange requires an internet connection and uses the HTTP or HTTPS protocol.

[0825] Specific system processing

[0826] Preparation before departure and image recognition

[0827] User: Before leaving the country, the user uses their smartphone to take pictures of their belongings. For example, they take pictures of the items in their suitcase one by one with the smartphone camera.

[0828] On the device: The captured image is temporarily stored in the device's storage, and then the device uploads the image to the server using the HTTP protocol.

[0829] Server: The server analyzes the received image data using TensorFlow and YOLO to automatically identify and classify the items. For example, it detects a "lighter" or "laptop" in the image. It compares the item with rules stored in a database to determine whether it is a prohibited or restricted item. The result of the determination is notified to the user's device.

[0830] Travel information collation and notification

[0831] Server: After receiving the image analysis results, the server compares the passport information with the regulations of the destination. For example, it can obtain the latest regulations of the destination through API and check whether the belongings violate the regulations. If any prohibited or restricted items are detected, the server will notify the user.

[0832] Device: The user device displays the notification received from the server, for example, a pop-up with specific instructions such as "Lighters are not allowed on board. Please put them in checked luggage."

[0833] Auto-generation of customs declaration documents

[0834] Server: The server automatically generates customs declaration forms based on the list of carried items. For example, it creates the declaration forms using a template engine such as "Jinja2." The generated forms are then sent to the user's device.

[0835] Terminal: The user terminal receives the automatically generated customs declaration form and displays it to the user, who then goes through customs procedures accordingly.

[0836] Smuggling detection and emotion recognition at immigration

[0837] Terminal: During immigration procedures, the user's terminal uses a camera and microphone to capture audio and video data, which is then transmitted to the server in real time. The emotion engine then recognizes the user's emotions from the audio and video data.

[0838] Server: The server performs analysis using emotion recognition engines such as Microsoft Azure Cognitive Services and IBM Watson. For example, it can detect user tension or anxiety. If suspicious behavior or negative emotions are detected, the server notifies security personnel in real time.

[0839] Emotion-based personalization

[0840] Server: The server provides personalized notifications and advice to users based on the emotion recognition results. For example, if tension is detected, the server sends a notification such as "Take a deep breath to relax."

[0841] Device: The user device receives this personalized notification and displays appropriate information to the user.

[0842] Examples of specific examples and prompts

[0843] Example 1: Before leaving the country, a user takes a picture of their baggage with their smartphone, and the device sends the picture to a server. If the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as checked baggage."

[0844] Example 2: If a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. The user then presents the form to the customs officer.

[0845] Example 3: If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If unnatural behavior is detected, the server immediately notifies security and additional measures are taken. If the emotion engine detects tension or anxiety in the user, the server sends a notification offering advice to the user on how to relax.

[0846] Example prompt sentence:

[0847] "Please take pictures of your belongings before you leave the country to make sure they do not contain any prohibited items."

[0848] "Auto-generate customs declaration documents that comply with the regulations of your destination."

[0849] "Detect smuggling and suspicious behavior in real time at the point of entry and provide security notifications based on emotion recognition results."

[0850] The system will reduce immigration issues, significantly improve the efficiency of security and customs procedures, and provide appropriate support based on the user's emotional state, ensuring a comfortable travel experience.

[0851] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0852] Step 1:

[0853] User: Before leaving the country, the user uses their smartphone to take a picture of their belongings.

[0854] Input: Image of belongings

[0855] Output: Image file saved in smartphone storage

[0856] Specific operation: The camera app on the smartphone is launched and each item is photographed individually. The photographed images are saved to the smartphone's internal storage or SD card.

[0857] Step 2:

[0858] On your device: Upload the saved image to the server.

[0859] Input: Image file saved in smartphone storage

[0860] Output: Image data sent to the server

[0861] What it does: It uses a smartphone app to send image data to a server via the HTTPS protocol, and displays a progress bar until all images taken by the user have been uploaded.

[0862] Step 3:

[0863] Server: Analyzes the received image data and identifies the item.

[0864] Input: Image data uploaded to the server

[0865] Output: List of identified items

[0866] How it works: The server runs deep learning models using TensorFlow and YOLO to identify the objects in each image, for example, determining whether the image contains a "lighter" or a "laptop."

[0867] Step 4:

[0868] Server: Determines whether the identified item is a prohibited or restricted item.

[0869] Input: List of identified items

[0870] Output: List of prohibited and restricted items

[0871] Specific operation: The system checks the regulations of the destination country stored in the database to determine whether the identified item is a prohibited or restricted item. For example, a "lighter" is detected as a prohibited item on board.

[0872] Step 5:

[0873] Server: Notifies the user of the result.

[0874] Input: List of prohibited and restricted items

[0875] Output: Notification message to the user's terminal

[0876] Specific operation: The server sends a push notification to the user's smartphone. For example, a message such as "Lighters are not permitted on board. Please check them in." is displayed.

[0877] Step 6:

[0878] Server: Automatically generate customs declaration documents.

[0879] Input: List of belongings

[0880] Output: Auto-generated customs declaration form

[0881] What happens: The server uses a template engine (e.g., Jinja2) to generate a customs declaration form from the list of items carried. The documents include details of the items that need to be declared.

[0882] Step 7:

[0883] Server: Sends automatically generated customs declaration documents to the user's terminal.

[0884] Input: Auto-generated customs declaration form

[0885] Output: Customs declaration document displayed on user's terminal

[0886] Specific operation: The server generates a customs declaration form, which is sent to the user's smartphone and displayed on the device. The user can save it as a PDF or present it directly to customs.

[0887] Step 8:

[0888] Device: During immigration procedures, a camera and microphone are used to capture audio and video data and transmit it to a server.

[0889] Input: Audio and video data

[0890] Output: Audio and video data sent to the server

[0891] Specific operation: During immigration procedures, the smartphone's camera and microphone are activated, and real-time audio and video recordings are sent to the server via the HTTPS protocol.

[0892] Step 9:

[0893] Server: Analyzes audio and video data and recognizes emotions.

[0894] Input: Audio and video data sent to the server

[0895] Output: Emotion recognition result

[0896] Specific operation: The server uses Microsoft Azure Cognitive Services and IBM Watson to analyze emotions from voice and recognize facial expressions from video to determine the user's emotional state.

[0897] Step 10:

[0898] Server: Notifies security personnel if suspicious behavior or negative sentiment is detected.

[0899] Input: Emotion recognition results

[0900] Output: Notification alert to security personnel

[0901] What it does: When abnormal behavior or negative sentiment is detected, the server sends real-time notifications to security personnel using a security alert system (e.g., PagerDuty).

[0902] Step 11:

[0903] Server: Provides personalized notifications and advice to users based on emotion recognition results.

[0904] Input: Emotion recognition results

[0905] Output: A personalized notification message

[0906] Specific operation: When a user feels nervous or anxious, the server generates an appropriate message and sends it to the user's smartphone, including advice such as "Take a deep breath to relax."

[0907] (Application example 2)

[0908] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0909] Immigration procedures, such as baggage inspections and customs procedures, are cumbersome and time-consuming, making it easy for fraud and violations to be overlooked. Furthermore, there is an insufficient system for recognizing suspicious passenger behavior and emotional states in real time and responding immediately, increasing security risks. These issues increase the burden on both travelers and inspectors, and create challenges for the overall travel experience.

[0910] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0911] In this invention, the server includes means for taking images of carried items using a user terminal, means for analyzing the taken images by the server and detecting prohibited or restricted items, means for notifying the user of the detection results, means for capturing audio and video data in real time and recognizing emotions using an emotion engine, and means for analyzing the recognition results of the emotion engine by the server to detect suspicious actions or behavior and notify security personnel in real time. This enables immigration procedures to be carried out quickly and accurately, and enables early detection and response to violations and fraudulent acts.

[0912] A "user terminal" is a device for taking pictures of personal items and capturing audio and video data.

[0913] A "server" is a central system that receives, analyzes, and processes data sent from user terminals.

[0914] "Personal effects" refers to all items that a user carries with them when entering or leaving a country.

[0915] "Means for taking images" refers to the function of capturing images of carried items using the camera of the user terminal.

[0916] "Prohibited or restricted items" are items that are prohibited or restricted from being carried on board by the airline or the country of destination.

[0917] "Means for detection" refers to the function of analyzing the image data received by the server and identifying prohibited or restricted items.

[0918] "Means for notifying" refers to a function that sends the detection results to the user terminal and prompts the user to take appropriate action.

[0919] "Means for capturing audio and video data in real time" refers to the ability to record environmental sounds and videos in real time using the camera and microphone of the user terminal.

[0920] "Emotion engine" refers to an analysis function for recognizing the user's emotional state from audio and video data.

[0921] "Suspicious behavior and speech" refers to abnormal behavior or speech that is not in line with normal patterns of behavior and is deemed to pose a security risk.

[0922] "Security Officer" refers to the personnel who conduct inspections, monitoring, and respond as necessary.

[0923] "Means for analyzing the recognition results of the emotion engine by the server" refers to a function for the server to perform behavioral analysis based on data obtained from the emotion engine.

[0924] "Means of notifying security personnel in real time" refers to the function of immediately sending an alert to security personnel when abnormal behavior or speech is detected.

[0925] The present invention is a system centered around a user terminal, a server, an emotion engine, and the exchange of data between them. The main components of this system and their functions are specifically shown below.

[0926] Hardware and software used

[0927] User device: Smartphone or dedicated device

[0928] Server: Cloud Server

[0929] Emotion Engine: EmotionEngine (emotion recognition module)

[0930] Image processing software: OpenCV

[0931] Server communication library: requests

[0932] Customs declaration generation software: TaxDeclarationGenerator (automatic generation module)

[0933] Data processing and calculation

[0934] 1. Taking and analyzing images of personal belongings

[0935] When a user arrives at the airport, they use the camera on their device to take a picture of their belongings. The captured image is temporarily saved on the device. The image is then encoded using OpenCV, an image processing software, and uploaded to the server. The server receives the uploaded image data and uses AI technology to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item and notifies the user's device of the result.

[0936] 2. Travel information collation and notification

[0937] The server compares the user's passport information with the regulations of the destination country based on the image analysis results. If a prohibited or restricted item is detected, the server notifies the user of the results and prompts the user to take necessary measures. For example, if a prohibited item such as a "lighter" is detected, the server notifies the user that "lighters are not permitted on board. Please check them in as luggage."

[0938] 3. Automatic generation of customs declaration documents

[0939] The server automatically generates a customs declaration form based on the list of recognized items. This process includes determining whether the items are subject to declaration. The automatically generated customs declaration form is sent to the user's terminal, where the user can use it for customs procedures.

[0940] 4. Emotion Recognition and Suspicious Behavior Detection at Entry

[0941] During immigration inspections, the user's device captures audio and video data in real time and sends it to the server. The emotion engine recognizes the user's emotions from this data, and the server analyzes the recognition results. If suspicious behavior or speech is detected, the server immediately notifies security personnel. For example, if the user's history is normal but significant tension or anxiety is detected, the server will transmit that information to security personnel in real time.

[0942] 5. Personalized notifications and relaxation guide

[0943] Based on the emotion recognition results, the server provides personalized notifications and advice to the user. For example, if tension or anxiety is detected, a notification offering guidance and voice advice on how to relax will be sent to the user's device. This reduces the user's stress and provides a comfortable travel experience.

[0944] Examples of concrete examples and prompts

[0945] Example: Security Assistant App Usage Scenario

[0946] When a user arrives at the airport, they take a photo of their belongings and upload it before checking in. The image is analyzed and a notification is sent to them that the detected "lighter" is not permitted. Later, if the user becomes nervous during immigration, the emotion engine detects this and the server notifies the security officer.

[0947] Prompt Sentence Examples

[0948] Consider a scenario where a user takes and uploads a picture of their baggage at an airport. The image is analyzed to classify the items according to airline and destination country regulations, detect any illegal items, and then recognize the user's emotions in real time, notifying security if suspicious behavior is detected.

[0949] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0950] Step 1:

[0951] The user arrives at the airport and takes a picture of their belongings using the camera on the user terminal.

[0952] Input: Image of belongings

[0953] Output: Captured image file

[0954] Specific operation: Capture an image of the item with the user's device's camera app and save it as an image file on the device.

[0955] Step 2:

[0956] The captured image files are encoded using OpenCV and uploaded to the server.

[0957] Input: Image file

[0958] Output: Send image data to the server

[0959] Specific operation: Encodes an image using the OpenCV library on the user device, and sends the image data to the server as an HTTP POST request using the requests library.

[0960] Step 3:

[0961] The server analyzes the received image data and uses AI technology to recognize and classify the items.

[0962] Input: Encoded image data

[0963] Output: Analysis results (list of recognized items, detection results of prohibited and restricted items)

[0964] How it works: The server uses an AI model to analyze image data, automatically classify recognized items, and generate detection results for prohibited and restricted items.

[0965] Step 4:

[0966] The server notifies the user of the analysis results and prompts the user to take necessary measures.

[0967] Input: Analysis results (detection results of prohibited and restricted items)

[0968] Output: Notification message to the user's terminal

[0969] Specific operation: The server generates a notification message based on the detection result and sends it to the user's device. For example, if a prohibited item such as a "lighter" is detected, the server notifies the user by saying, "Lighters are prohibited from being carried on board. Please put them in checked baggage."

[0970] Step 5:

[0971] The server compares the analysis results with travel information to check whether the items being carried are in compliance with regulations.

[0972] Input: Analysis results, travel information (passport information)

[0973] Output: Whether or not there is a violation and a notification message

[0974] Specific operation: The server compares the travel information with the analysis results, and if any illegal items are found, sends a message to the user's device.

[0975] Step 6:

[0976] During immigration control, the user terminal captures audio and video data in real time and transmits it to the server.

[0977] Input: Real-time audio and video data

[0978] Output: Sending audio and video data to the server

[0979] Specific operation: Captures environmental sounds and videos in real time using the camera and microphone of the user device and sends them to the server.

[0980] Step 7:

[0981] The server analyzes the received audio and video data using an emotion engine to recognize the user's emotions.

[0982] Input: Real-time audio and video data

[0983] Output: Emotion recognition result

[0984] Specific operation: The server uses EmotionEngine to analyze audio and video data and recognize the user's emotional state.

[0985] Step 8:

[0986] The server analyzes the emotion recognition results and notifies security personnel if any suspicious behavior or speech is detected.

[0987] Input: Emotion recognition results

[0988] Output: A message to notify the security officer

[0989] Specific operation: The server analyzes the emotion recognition results, and if any suspicious behavior or speech is detected, it sends a message to the security officer in real time.

[0990] Step 9:

[0991] The server provides personalized notifications and advice to the user based on the emotion recognition results.

[0992] Input: Emotion recognition results

[0993] Output: A personalized notification message to the user

[0994] Specific operation: If the server detects tension or anxiety based on the emotion recognition results, it sends a notification message to the user's device containing relaxation guidance and advice.

[0995] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0996] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0997] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0998] [Third embodiment]

[0999] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1000] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1001] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1002] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1003] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1004] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1005] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1006] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1007] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1008] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1009] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1010] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1011] The embodiment of the present invention will be described with a focus on a user terminal, a server, and data exchange between them.

[1012] Advance preparation before departure and image recognition

[1013] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[1014] Device: The captured images are temporarily stored on the device, which then uploads them to the server.

[1015] Server: The server analyzes the received image data. During the analysis, AI technology is used to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item. The result of the determination is sent to the user's device.

[1016] Travel information collation and notification

[1017] Server: After obtaining the image analysis results, the server checks the user's passport information. Specifically, it compares the user's belongings with the destination country's regulations and airline regulations to see if they violate any rules. If prohibited or restricted items are detected, the server notifies the user.

[1018] Terminal: The user terminal displays the notification received from the server and prompts the user to take necessary measures, such as changing prohibited items into checked baggage.

[1019] Auto-generation of customs declaration documents

[1020] Server: The server automatically generates the necessary customs declaration forms based on the list of recognized items, including determining whether the items require declaration.

[1021] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed to the user on the terminal, allowing the user to follow the instructions and prepare for a smooth customs process.

[1022] Smuggling detection at the point of entry

[1023] Device: During immigration procedures, the user's device captures audio and video data and transmits it to a server in real time, allowing the user's actions and behavior during the procedure to be monitored.

[1024] Server: The server analyzes the received audio and video data to detect suspicious behavior or actions, such as when a user makes contradictory statements or exhibits specific non-verbal behavior.

[1025] Server: When suspicious activity is detected, the server notifies security personnel in real time, allowing for any necessary further investigations or responses to be taken promptly.

[1026] Specific examples

[1027] Example 1: Image recognition before departure

[1028] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[1029] Example 2: Automatic generation of customs declaration documents

[1030] When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. By presenting this form at the customs office, the user can smoothly proceed with customs clearance.

[1031] Example 3: Detecting contraband

[1032] If a user behaves suspiciously during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security, and additional measures are taken.

[1033] This system will reduce immigration issues and significantly improve the efficiency of security and customs inspections.

[1034] The processing flow will be explained below.

[1035] Step 1:

[1036] User: Takes pictures of their belongings using a smartphone or dedicated device. The user takes pictures of all items in turn and records the necessary information.

[1037] Step 2:

[1038] Device: Temporarily stores captured images. Once all items have been photographed, the image data will be ready to upload.

[1039] Step 3:

[1040] On the device: Upload the captured image to the server. Display a notification to the user when the upload is complete.

[1041] Step 4:

[1042] Server: Analyzes the received image data. Using AI technology, it recognizes and classifies the objects in each image.

[1043] Step 5:

[1044] Server: Compares the analysis results with the destination country's regulations and airline rules to identify prohibited or restricted items.

[1045] Step 6:

[1046] Server: If prohibited or restricted items are detected, it generates a list of them and creates notification data.

[1047] Step 7:

[1048] Terminal: Receives notification data sent from the server and displays it to the user. The user checks the displayed information and takes necessary measures.

[1049] Step 8:

[1050] Server: Based on the list of recognized items, a customs declaration form is automatically generated, listing the items that must be declared.

[1051] Step 9:

[1052] Server: Sends automatically generated customs declaration documents to the user's terminal.

[1053] Step 10:

[1054] Terminal: Displays the customs declaration documents sent to the user's terminal. The user checks the contents of the documents and prepares for customs procedures.

[1055] Step 11:

[1056] Terminal: During immigration procedures, audio and video data is captured and transmitted to a server in real time.

[1057] Step 12:

[1058] Server: Analyzes received audio and video data to detect suspicious behavior and speech, including non-verbal behavior and contradictory statements.

[1059] Step 13:

[1060] Server: If suspicious activity is detected, it notifies security personnel in real time and sends the analysis results as additional information.

[1061] Step 14:

[1062] Security personnel: Receive notifications from the server and promptly carry out any necessary additional checks or actions.

[1063] Example 1

[1064] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1065] Conventional immigration control systems require users to perform numerous manual tasks when handling carry-on baggage and customs declaration procedures, resulting in inefficiency and a high likelihood of errors. Furthermore, detecting smuggling during immigration inspections relies on human resources, making thorough monitoring difficult. Therefore, there is a need for a system that can improve efficiency and prevent fraud in each process of carry-on baggage management, customs declaration, and immigration inspection.

[1066] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1067] In this invention, the server includes means for taking images of carried items using a user terminal, means for temporarily saving the images taken by the terminal, means for uploading the images saved by the terminal to the server, means for analyzing the image data received by the server using AI technology to recognize and classify the items, means for determining whether the items fall under prohibited or restricted items, and means for transmitting the determination result to the user terminal. This automates the management of carried items, reduces the burden on users, and enables the prevention of fraudulent activities and the efficiency of immigration procedures.

[1068] A "user terminal" is an electronic device such as a mobile information terminal or a smartphone that is intended to be operated by a user.

[1069] "Personal effects" refers to all items that a user carries with them when traveling or moving.

[1070] The "means for taking an image" is a function for taking an image of a carried item using a camera function provided in the user terminal.

[1071] The "temporary storage means" is a function for storing captured images in a temporary storage device within the user terminal.

[1072] A "server" is a remote data processing device that receives and processes data sent from a user terminal.

[1073] "Uploading means" is a function for transmitting data from a user terminal to a server.

[1074] "Means for analyzing image data using AI technology" refers to a processing function that uses AI technology on a server to identify and classify objects contained in images.

[1075] "Means for recognizing and classifying items" refers to a function that identifies and classifies the name and type of an item from image data analyzed by AI.

[1076] "Means for determining whether a recognized item is a prohibited or restricted item" is a function that determines whether the recognized item violates the laws and regulations of the destination country.

[1077] The "means for transmitting the determination result to the user terminal and notifying the user" is a function for transmitting the determination result to the user terminal and displaying a notification to the user.

[1078] "Travel information" refers to all information related to a user's travel, such as passport information and airline ticket information.

[1079] "Means of matching" refers to the function of comparing the image analysis results with travel information to check for matches or violations.

[1080] The "means for automatically generating customs declaration documents" is a function that automatically creates the necessary customs declaration documents based on the analyzed carry-on item data.

[1081] The "means for displaying the automatically generated customs declaration document on the user terminal" is a function for transmitting the generated customs declaration document to the user terminal so that the user can view it on the terminal.

[1082] The present invention relates to a system for improving the efficiency of carry-on baggage management and customs declaration procedures at the time of immigration and for strengthening smuggling detection. Specific embodiments of the system are described below.

[1083] Preparation before departure and image recognition

[1084] User: Before leaving the country, the user takes pictures of their belongings using their smartphone or a dedicated device. This is done using the smartphone's camera function. For example, the user can take pictures of each item they put in their suitcase, obtaining image data of all of their belongings.

[1085] On the device: The captured images are saved in a temporary folder (local storage) on the user's device. They are then automatically uploaded from the device to the server. This upload process is carried out via a dedicated application on the device.

[1086] Server: The server analyzes the received image data using AI technology. Specifically, it uses image analysis libraries such as TensorFlow and OpenCV to recognize and classify objects. For example, objects in the image are recognized as "lighter," "bottle," "book," etc.

[1087] Server: Based on the analysis results, the server determines whether the item is a prohibited or restricted item. This involves checking it against a list of prohibited items stored on the server. The result of the determination is then sent to the user's device. The user is notified and can take the necessary measures accordingly.

[1088] Travel information collation and notification

[1089] Server: After obtaining the image analysis results, the server checks the user's passport information (electronic passport data) and travel information to ensure that the belongings comply with the regulations of the destination country and airline.

[1090] Server: If a prohibited or restricted item is detected, the server sends that information to the user's device and notifies the user of the appropriate action to take, such as "Lighters are not permitted on board. Please put them in checked baggage."

[1091] Auto-generation of customs declaration documents

[1092] Server: The server automatically generates the necessary customs forms based on the list of recognized items, including determining whether the items require declaration.

[1093] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed in a PDF viewer or dedicated application, allowing the user to follow the instructions and prepare for smooth customs procedures.

[1094] Smuggling detection at the point of entry

[1095] Terminal: During immigration control, the user's terminal captures audio and video data and transmits it to the server in real time using the terminal's built-in camera and microphone.

[1096] Server: The server analyzes the received audio and video data to detect suspicious behavior and speech. For example, it uses voice recognition and facial recognition systems to analyze the user's speech and non-verbal behavior (e.g., facial expressions and body movements).

[1097] Server: If any suspicious activity is detected, the server immediately notifies security personnel and prompts them to take any necessary further investigations or actions, thereby monitoring potential smuggling activity in real time.

[1098] Specific examples

[1099] Example 1: Before leaving the country, a user takes a picture of their baggage with their smartphone, and the device sends the picture to a server. If the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as checked baggage."

[1100] Example 2: When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. The user then presents the form at the customs office and the customs clearance process goes smoothly.

[1101] Example 3: If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security, and additional measures are taken.

[1102] This system will reduce immigration issues and significantly improve the efficiency of security and customs inspections.

[1103] Prompt Sentence Examples

[1104] Example: "How can I take a picture of my baggage before leaving the country and upload it to a server, so that AI can recognize and classify the items and notify the user if any prohibited or restricted items are detected?"

[1105] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1106] Step 1:

[1107] User: The user uses a smartphone or dedicated device to take pictures of each item in their baggage. Specifically, the user launches a camera app and takes pictures of each item of baggage.

[1108] Input: The physical state of the baggage.

[1109] Output: Digital image data.

[1110] Step 2:

[1111] Device: The captured image is automatically saved to a temporary folder on the user's device.

[1112] Input: Images taken with a camera app on a smartphone or dedicated device.

[1113] Output: Image file saved in the device's local storage.

[1114] Step 3:

[1115] On the device: To upload the saved image to the server, send the image data via the API. Specifically, send the image data to the server using an HTTP request.

[1116] Input: An image file stored in local storage.

[1117] Output: Image data sent to the server.

[1118] Step 4:

[1119] Server: The server analyzes the received image data using AI technology, for example, using image analysis libraries such as TensorFlow or OpenCV, and applies a model to recognize objects in the image.

[1120] Input: Image data sent to the server.

[1121] Output: Identified item information as a result of analysis (e.g. lighter, plastic bottle, book, etc.).

[1122] Step 5:

[1123] Server: Based on the analysis results, it determines whether the item is a prohibited or restricted item, and in this process, it checks against the prohibited items list.

[1124] Input: Identified item information as a result of analysis.

[1125] Output: Judgment result (e.g. prohibited item, restricted item, permitted item).

[1126] Step 6:

[1127] Server: Sends the judgment result to the user device. Specifically, it generates JSON data in the form of a notification and sends it to the user device.

[1128] Input: The result of the test.

[1129] Output: Notification data sent to the user device.

[1130] Step 7:

[1131] Terminal: The user terminal analyzes the received notification data and displays it to the user. Specifically, it uses a pop-up notification or an alert dialog to alert the user.

[1132] Input: Notification data sent by the server.

[1133] Output: The notification displayed on the user's screen.

[1134] Step 8:

[1135] Server: Compares the user's passport and travel information with the image analysis results, specifically by accessing a database and comparing them with the destination country's regulations and airline regulations.

[1136] Input: Passport information, travel information, and image analysis results.

[1137] Output: Verification data as a result of the match (e.g., whether or not there is a violation).

[1138] Step 9:

[1139] Server: If prohibited or restricted items are detected based on the matching results, the server notifies the user of this information.

[1140] Input: Verification data as a match result.

[1141] Output: The alert data sent to the user terminal.

[1142] Step 10:

[1143] Terminal: The user's terminal analyzes the received warning data and notifies the user of countermeasures. For example, it displays specific instructions such as "Lighters are prohibited from being carried on board. Please put them in checked baggage."

[1144] Input: The alert data sent by the server.

[1145] Output: The instructions displayed on the user's screen.

[1146] Step 11:

[1147] Server: Automatically generates customs declaration documents based on the list of recognized carry-on items. Specifically, it automatically fills in the necessary information in a template and generates the documents in PDF format.

[1148] Input: A list of recognized items.

[1149] Output: Auto-generated customs declaration form (PDF).

[1150] Step 12:

[1151] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed for the user to view.

[1152] Input: Auto-generated customs declaration form (PDF).

[1153] Output: Customs declaration document displayed on user's terminal.

[1154] Step 13:

[1155] Terminal: During immigration control, the user terminal captures audio and video data and transmits it to the server in real time using the built-in camera and microphone.

[1156] Input: Audio and video data from immigration inspection.

[1157] Output: Audio and video data sent to the server.

[1158] Step 14:

[1159] Server: The server analyzes the received audio and video data and detects suspicious behavior or speech. Specifically, it uses voice recognition and facial recognition systems.

[1160] Input: Audio and video data sent to the server.

[1161] Output: Evaluation data of behavior and speech as the analysis result.

[1162] Step 15:

[1163] Server: If suspicious activity is detected, the server notifies security personnel in real time, facilitating further investigation and response.

[1164] Input: Evaluation data of behavior and speech as the analysis results.

[1165] Output: Alert data sent to security personnel.

[1166] This processing step reduces immigration issues and significantly improves the efficiency of security and customs inspections.

[1167] (Application example 1)

[1168] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1169] When traveling internationally, it is cumbersome to check whether your belongings comply with regulations before leaving the country, and it takes time to fill out customs declaration forms. Furthermore, there is a lack of means to quickly identify travelers exhibiting suspicious behavior during immigration inspections, making it difficult to ensure security. There is a need for a solution to these problems, making immigration procedures more efficient and improving security.

[1170] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1171] In this invention, the server includes means for taking images of carried items using a user terminal, means for analyzing the images taken by the server to detect prohibited or restricted items, means for notifying the user of the detection results, means for automatically generating customs declaration forms and displaying them on the user terminal, means for capturing audio and video data during immigration inspection and transmitting it to the server in real time, means for analyzing the audio and video data in the server to detect suspicious behavior, and means for notifying security personnel when suspicious behavior is detected. This enables quick confirmation of carried items before departure, automates the creation of customs declaration forms, and strengthens security inspections at the time of entry.

[1172] A "user terminal" is an electronic device that is carried and operated by a user, and is a device for taking images of carried items and receiving notifications.

[1173] A "server" is a computer system connected to a network that stores, analyzes, and processes data.

[1174] "Personal effects" are items that a user carries with them when traveling or going out.

[1175] The "means for taking an image" is a mechanism for taking a photo of a carried item using the camera function installed in the user terminal.

[1176] The "means for analyzing images" refers to algorithms or software for processing the image data received by the server and detecting specific items.

[1177] "Prohibited items" are items that are prohibited from being possessed or brought into the country under laws and regulations.

[1178] "Restricted items" are items whose possession or import is restricted based on permits or conditions.

[1179] "Means for notifying" refers to a technology for transmitting information from a server to a user terminal in real time and informing the user.

[1180] "Customs Declaration Form" means an official document used to declare possession of goods to customs.

[1181] "Automatic generation means" refers to a system that uses a server algorithm to automatically generate the necessary documents.

[1182] "Means for capturing audio and video data" refers to the process of acquiring audio and video using a microphone and camera installed on the user terminal.

[1183] "Means for transmitting in real time" refers to Internet communication technology for instantly transmitting data to a server.

[1184] "Means for detecting suspicious behavior" refers to technology that automatically detects suspicious behavior and speech based on audio and video data on the server.

[1185] "Means for notifying security personnel" refers to a system for sending warning information to security personnel when the server detects suspicious activity.

[1186] The present invention provides a system for improving the efficiency of baggage management before departure and security checks at the time of entry through data exchange between a user terminal and a server. Specific embodiments of the system are described below.

[1187] Baggage management before departure

[1188] 1. Taking pictures using a user device

[1189] A user takes a picture of their belongings with a user terminal such as a smartphone, and image data of the baggage is acquired using the camera of the smartphone.

[1190] 2. Sending image data

[1191] The captured image data is uploaded from the user's device to a server via an internet connection, and the data is sent in real time.

[1192] 3. Server Image Analysis

[1193] The server analyzes the received image data using AI technology. Specifically, it uses an object recognition algorithm to identify items in the image and detect prohibited or restricted items. For this, image analysis software such as OpenCV or TensorFlow can be used.

[1194] 4. Notification of test results

[1195] The analysis results are sent to the user's device, where the user can check whether or not there are any prohibited or restricted items. If the user accidentally carries a prohibited item, the device will notify the user in real time.

[1196] 5. Automatic generation of customs declaration documents

[1197] The server automatically generates customs declaration forms based on the analyzed list of carried items, and the generated forms are sent to the user's terminal, where the user can view and print them.

[1198] Security checks at entry

[1199] 1. Audio and Video Capture

[1200] During immigration inspection, the user's device (smartphone) captures audio and video data in real time and transmits it to a server using the smartphone's microphone and camera.

[1201] 2. Real-time data analysis

[1202] The server analyzes the received audio and video data, using natural language processing technology for audio analysis and object detection and motion analysis technology for video analysis, to detect suspicious behavior or contradictory statements in real time.

[1203] 3. Security Notice

[1204] If any suspicious activity is detected, the server immediately notifies security personnel, allowing any necessary additional testing or response to be carried out promptly.

[1205] Examples of specific examples and prompts

[1206] Example 1: Image recognition before departure

[1207] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[1208] Example 2: Security checks at entry

[1209] If a user behaves suspiciously during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security personnel, who then take additional action.

[1210] Prompt for the generative AI model (luggage analysis example):

[1211] "Analyze the image to determine if it contains any illegal items. The image data is contained in the next argument, image_data."

[1212] The above embodiment is expected to improve the efficiency of immigration procedures and security.

[1213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1214] Step 1:

[1215] A user takes a photo of their belongings with a user terminal such as a smartphone. The captured image data is acquired using the smartphone's camera. The input is the image data captured by the user's operation, and the output is the image data temporarily saved on the terminal.

[1216] Step 2:

[1217] The device uploads the captured image data to a server. The data is transmitted in real time via an internet connection. The input is the captured image data, and the output is the image data transmitted to the server.

[1218] Step 3:

[1219] The server analyzes the received image data using AI technology. Specifically, it uses object recognition algorithms (e.g., OpenCV or TensorFlow) to identify items in the image and detect prohibited or restricted items. The input is the image data sent to the server, and the output is the analysis results.

[1220] Step 4:

[1221] The server sends a notification to the user's device based on the analysis results. The notification includes details of the prohibited or restricted items detected. The input is the result data from the AI ​​analysis, and the output is a notification message sent to the user's device.

[1222] Step 5:

[1223] The server automatically generates customs declaration forms based on the analyzed list of carried items, including determining which items should be declared. The input is the list of items based on the analysis results, and the output is the generated customs declaration form.

[1224] Step 6:

[1225] The server sends the automatically generated customs declaration form to the user's terminal, where the user can check and print it. The input is the generated customs declaration form, and the output is the declaration form sent to the user's terminal.

[1226] Step 7:

[1227] During immigration control, the user's device (smartphone) captures audio and video data and transmits it to the server in real time. The data is acquired using the smartphone's microphone and camera. The input is the audio and video data captured at the immigration control site, and the output is the audio and video data transmitted to the server.

[1228] Step 8:

[1229] The server analyzes the received audio and video data. It uses natural language processing technology for audio analysis and object detection and motion analysis technology for video analysis. The input is the captured audio and video data, and the output is behavioral judgment data as the analysis result.

[1230] Step 9:

[1231] Based on the analysis results, the server determines whether there is any suspicious behavior or contradictory statements, and notifies security personnel if any suspicious behavior is detected. The input is behavior determination data, and the output is a notification message to the security personnel.

[1232] These steps ensure efficient and effective baggage checks before departure, automatic generation of customs declaration forms, and security checks upon entry.

[1233] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1234] In the present invention, a specific embodiment for implementing a system centered on a user terminal, a server, an emotion engine, and data exchange between these will be described.

[1235] Advance preparation before departure and image recognition

[1236] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[1237] Device: The captured images are temporarily stored on the device, which then uploads them to the server.

[1238] Server: The server analyzes the received image data. It uses AI technology to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item. The result of the determination is sent to the user's device.

[1239] Travel information collation and notification

[1240] Server: After obtaining the image analysis results, the server checks the user's passport information. Specifically, it checks the user's belongings against the regulations of the destination country and the airline to see if they violate any rules. If any prohibited or restricted items are detected, the server notifies the user.

[1241] Terminal: The user terminal displays the notification received from the server and prompts the user to take necessary measures, such as changing prohibited items into checked baggage.

[1242] Auto-generation of customs declaration documents

[1243] Server: The server automatically generates the necessary customs declaration forms based on the list of recognized items, including determining whether the items require declaration.

[1244] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed to the user on the terminal, allowing the user to follow the instructions and prepare for a smooth customs process.

[1245] Smuggling detection and emotion recognition at immigration

[1246] Terminal: During immigration procedures, the user's terminal captures audio and video data and transmits it to the server in real time. The emotion engine also recognizes the user's emotions from the audio and video data.

[1247] Server: The server analyzes the received audio and video data and the emotion recognition results obtained from the emotion engine to detect suspicious behavior or speech. For example, it determines if the user makes contradictory statements or exhibits specific non-verbal behavior.

[1248] Server: When suspicious behavior or negative sentiment is detected, the server notifies security personnel in real time, allowing for any necessary further investigations or responses to be taken promptly.

[1249] Emotion-based personalization

[1250] Server: Based on the emotion recognition results obtained from the emotion engine, the server provides personalized notifications and advice to the user. For example, if tension or anxiety is detected, the server provides guidance and advice on how to relax.

[1251] Device: The user's device receives this personalized notification and provides the user with appropriate information to support a comfortable travel experience.

[1252] Specific examples

[1253] Example 1: Image recognition before departure

[1254] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[1255] Example 2: Automatic generation of customs declaration documents

[1256] When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. By presenting this form at the customs office, the user can smoothly proceed with customs clearance.

[1257] Example 3: Contraband detection and emotion recognition

[1258] If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If unnatural behavior is detected, the server immediately notifies security, and additional measures are taken. If the emotion engine detects tension or anxiety in the user, the server sends a notification offering advice to the user on how to relax.

[1259] The system will reduce immigration issues, significantly improve the efficiency of security and customs inspections, and provide appropriate support based on the user's emotional state, ensuring a comfortable travel experience.

[1260] The processing flow will be explained below.

[1261] Step 1:

[1262] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[1263] Step 2:

[1264] Device: Images are temporarily stored on the device. Once all items have been photographed, the image data will be ready to upload.

[1265] Step 3:

[1266] On the device: Upload the captured image to the server. Display a notification to the user when the upload is complete.

[1267] Step 4:

[1268] Server: Analyzes the received image data. Using AI technology, it recognizes and classifies the objects in each image.

[1269] Step 5:

[1270] Server: Compares the analysis results with the destination country's regulations and airline rules to identify prohibited or restricted items.

[1271] Step 6:

[1272] Server: If prohibited or restricted items are detected, it generates a list of them and creates notification data.

[1273] Step 7:

[1274] Terminal: Receives notification data sent from the server and displays it to the user. The user checks the displayed information and takes necessary measures.

[1275] Step 8:

[1276] Server: Based on the list of recognized items, a customs declaration form is automatically generated, listing the items that must be declared.

[1277] Step 9:

[1278] Server: Sends automatically generated customs declaration documents to the user's terminal.

[1279] Step 10:

[1280] Terminal: Displays the customs declaration documents sent to the user's terminal. The user checks the contents of the documents and prepares for customs procedures.

[1281] Step 11:

[1282] Terminal: During immigration procedures, audio and video data is captured and sent to the server in real time. An emotion engine also runs to recognize the user's emotions.

[1283] Step 12:

[1284] Server: Analyzes the received audio and video data and emotion recognition results obtained from the emotion engine to detect suspicious behavior or speech. For example, it determines if the user makes contradictory statements or engages in specific non-verbal behavior.

[1285] Step 13:

[1286] Server: When suspicious behavior or negative sentiment is detected, security personnel are notified in real time, allowing for any necessary further investigation or response to be taken promptly.

[1287] Step 14:

[1288] Server: Provides personalized notifications and advice to users based on the emotion recognition results obtained from the emotion engine. For example, if tension or anxiety is detected, it sends guidance and advice on how to relax.

[1289] Step 15:

[1290] Device: The user's device receives the personalized notification and displays appropriate information to the user, who can then check the notification and take action to relax.

[1291] Example 2

[1292] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1293] In recent years, with the increase in cross-border travel, safety and efficiency at immigration are required, but current systems have difficulty in detecting the bringing in of prohibited or restricted items in advance. Another issue is that customs declaration procedures and security checks at the point of entry require time and effort from users. This reduces convenience for travelers and the efficiency of security and customs inspections.

[1294] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for taking an image of the carried items using the user terminal, a means for temporarily saving the image taken by the terminal and uploading it to the server, a means for analyzing the image data received by the server and classifying the identified items, a means for determining whether the identified items fall under prohibited or restricted items, and a means for notifying the user terminal of the determination result. This enables the user to quickly check the carried items before departure and take necessary measures. In addition, customs declaration procedures are simplified and security checks at the time of entry are made more efficient.

[1295] A "user terminal" is an electronic device that a user uses to take pictures of their belongings and upload the data to a server.

[1296] The "means for taking an image" is a function that allows a user to capture an image of an item carried by the user using the user terminal.

[1297] "Terminal" refers to a user terminal or a device that has the function of temporarily storing captured images and uploading them to a server.

[1298] The "means for temporarily saving images" is a function for temporarily saving captured images on the terminal.

[1299] The "server" is a central management system that analyzes the received image data, classifies the identified items, and notifies the results to the user terminal.

[1300] The "means for analyzing image data" refers to algorithms or software used by the server to analyze the image data received.

[1301] The "means for classifying identified items" is a function by which the server identifies items from image data and classifies them into categories.

[1302] The "means of determining whether an item is a prohibited or restricted item" is a function that uses analyzed data to check whether the identified item violates the regulations of the destination or airline.

[1303] The "means for notifying the user terminal of the determination result" is a function by which the server sends a message to the user terminal to notify the user of the determination result.

[1304] "Travel information" is data such as the user's passport information and travel destination information.

[1305] The "means for automatically generating customs declaration documents" is a function that allows the server to automatically generate customs declaration documents based on the list of carried items.

[1306] The present invention provides a system for travelers to efficiently and safely complete immigration procedures. The system includes a user terminal, a server, and a network for exchanging data between them.

[1307] Hardware and software used

[1308] User device: A device such as a smartphone or tablet that takes pictures of personal items and uploads the data to a server. This device must have a camera and internet connection.

[1309] Server: The server is a central management system that receives and analyzes image data. Deep learning frameworks such as "TensorFlow" and "YOLO" are used for analysis. MySQL and PostgreSQL are used as databases.

[1310] Network: Data exchange requires an internet connection and uses the HTTP or HTTPS protocol.

[1311] Specific system processing

[1312] Preparation before departure and image recognition

[1313] User: Before leaving the country, the user uses their smartphone to take pictures of their belongings. For example, they take pictures of the items in their suitcase one by one with the smartphone camera.

[1314] On the device: The captured image is temporarily stored in the device's storage, and then the device uploads the image to the server using the HTTP protocol.

[1315] Server: The server analyzes the received image data using TensorFlow and YOLO to automatically identify and classify the items. For example, it detects a "lighter" or "laptop" in the image. It compares the item with rules stored in a database to determine whether it is a prohibited or restricted item. The result of the determination is notified to the user's device.

[1316] Travel information collation and notification

[1317] Server: After receiving the image analysis results, the server compares the passport information with the regulations of the destination. For example, it can obtain the latest regulations of the destination through API and check whether the belongings violate the regulations. If any prohibited or restricted items are detected, the server will notify the user.

[1318] Device: The user device displays the notification received from the server, for example, a pop-up with specific instructions such as "Lighters are not allowed on board. Please put them in checked luggage."

[1319] Auto-generation of customs declaration documents

[1320] Server: The server automatically generates customs declaration forms based on the list of carried items. For example, it creates the declaration forms using a template engine such as "Jinja2." The generated forms are then sent to the user's device.

[1321] Terminal: The user terminal receives the automatically generated customs declaration form and displays it to the user, who then goes through customs procedures accordingly.

[1322] Smuggling detection and emotion recognition at immigration

[1323] Terminal: During immigration procedures, the user's terminal uses a camera and microphone to capture audio and video data, which is then transmitted to the server in real time. The emotion engine then recognizes the user's emotions from the audio and video data.

[1324] Server: The server performs analysis using emotion recognition engines such as Microsoft Azure Cognitive Services and IBM Watson. For example, it can detect user tension or anxiety. If suspicious behavior or negative emotions are detected, the server notifies security personnel in real time.

[1325] Emotion-based personalization

[1326] Server: The server provides personalized notifications and advice to users based on the emotion recognition results. For example, if tension is detected, the server sends a notification such as "Take a deep breath to relax."

[1327] Device: The user device receives this personalized notification and displays appropriate information to the user.

[1328] Examples of specific examples and prompts

[1329] Example 1: Before leaving the country, a user takes a picture of their baggage with their smartphone, and the device sends the picture to a server. If the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as checked baggage."

[1330] Example 2: If a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. The user then presents the form to the customs officer.

[1331] Example 3: If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If unnatural behavior is detected, the server immediately notifies security and additional measures are taken. If the emotion engine detects tension or anxiety in the user, the server sends a notification offering advice to the user on how to relax.

[1332] Example prompt sentence:

[1333] "Please take pictures of your belongings before you leave the country to make sure they do not contain any prohibited items."

[1334] "Auto-generate customs declaration documents that comply with the regulations of your destination."

[1335] "Detect smuggling and suspicious behavior in real time at the point of entry and provide security notifications based on emotion recognition results."

[1336] The system will reduce immigration issues, significantly improve the efficiency of security and customs procedures, and provide appropriate support based on the user's emotional state, ensuring a comfortable travel experience.

[1337] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1338] Step 1:

[1339] User: Before leaving the country, the user uses their smartphone to take a picture of their belongings.

[1340] Input: Image of belongings

[1341] Output: Image file saved in smartphone storage

[1342] Specific operation: The camera app on the smartphone is launched and each item is photographed individually. The photographed images are saved to the smartphone's internal storage or SD card.

[1343] Step 2:

[1344] On your device: Upload the saved image to the server.

[1345] Input: Image file saved in smartphone storage

[1346] Output: Image data sent to the server

[1347] What it does: It uses a smartphone app to send image data to a server via the HTTPS protocol, and displays a progress bar until all images taken by the user have been uploaded.

[1348] Step 3:

[1349] Server: Analyzes the received image data and identifies the item.

[1350] Input: Image data uploaded to the server

[1351] Output: List of identified items

[1352] How it works: The server runs deep learning models using TensorFlow and YOLO to identify the objects in each image, for example, determining whether the image contains a "lighter" or a "laptop."

[1353] Step 4:

[1354] Server: Determines whether the identified item is a prohibited or restricted item.

[1355] Input: List of identified items

[1356] Output: List of prohibited and restricted items

[1357] Specific operation: The system checks the regulations of the destination country stored in the database to determine whether the identified item is a prohibited or restricted item. For example, a "lighter" is detected as a prohibited item on board.

[1358] Step 5:

[1359] Server: Notifies the user of the result.

[1360] Input: List of prohibited and restricted items

[1361] Output: Notification message to the user's terminal

[1362] Specific operation: The server sends a push notification to the user's smartphone. For example, a message such as "Lighters are not permitted on board. Please check them in." is displayed.

[1363] Step 6:

[1364] Server: Automatically generate customs declaration documents.

[1365] Input: List of belongings

[1366] Output: Auto-generated customs declaration form

[1367] What happens: The server uses a template engine (e.g., Jinja2) to generate a customs declaration form from the list of items carried. The documents include details of the items that need to be declared.

[1368] Step 7:

[1369] Server: Sends automatically generated customs declaration documents to the user's terminal.

[1370] Input: Auto-generated customs declaration form

[1371] Output: Customs declaration document displayed on user's terminal

[1372] Specific operation: The server generates a customs declaration form, which is sent to the user's smartphone and displayed on the device. The user can save it as a PDF or present it directly to customs.

[1373] Step 8:

[1374] Device: During immigration procedures, a camera and microphone are used to capture audio and video data and transmit it to a server.

[1375] Input: Audio and video data

[1376] Output: Audio and video data sent to the server

[1377] Specific operation: During immigration procedures, the smartphone's camera and microphone are activated, and real-time audio and video recordings are sent to the server via the HTTPS protocol.

[1378] Step 9:

[1379] Server: Analyzes audio and video data and recognizes emotions.

[1380] Input: Audio and video data sent to the server

[1381] Output: Emotion recognition result

[1382] Specific operation: The server uses Microsoft Azure Cognitive Services and IBM Watson to analyze emotions from voice and recognize facial expressions from video to determine the user's emotional state.

[1383] Step 10:

[1384] Server: Notifies security personnel if suspicious behavior or negative sentiment is detected.

[1385] Input: Emotion recognition results

[1386] Output: Notification alert to security personnel

[1387] What it does: When abnormal behavior or negative sentiment is detected, the server sends real-time notifications to security personnel using a security alert system (e.g., PagerDuty).

[1388] Step 11:

[1389] Server: Provides personalized notifications and advice to users based on emotion recognition results.

[1390] Input: Emotion recognition results

[1391] Output: A personalized notification message

[1392] Specific operation: When a user feels nervous or anxious, the server generates an appropriate message and sends it to the user's smartphone, including advice such as "Take a deep breath to relax."

[1393] (Application example 2)

[1394] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1395] Immigration procedures, such as baggage inspections and customs procedures, are cumbersome and time-consuming, making it easy for fraud and violations to be overlooked. Furthermore, there is an insufficient system for recognizing suspicious passenger behavior and emotional states in real time and responding immediately, increasing security risks. These issues increase the burden on both travelers and inspectors, and create challenges for the overall travel experience.

[1396] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1397] In this invention, the server includes means for taking images of carried items using a user terminal, means for analyzing the taken images by the server and detecting prohibited or restricted items, means for notifying the user of the detection results, means for capturing audio and video data in real time and recognizing emotions using an emotion engine, and means for analyzing the recognition results of the emotion engine by the server to detect suspicious actions or behavior and notify security personnel in real time. This enables immigration procedures to be carried out quickly and accurately, and enables early detection and response to violations and fraudulent acts.

[1398] A "user terminal" is a device for taking pictures of personal items and capturing audio and video data.

[1399] A "server" is a central system that receives, analyzes, and processes data sent from user terminals.

[1400] "Personal effects" refers to all items that a user carries with them when entering or leaving a country.

[1401] "Means for taking images" refers to the function of capturing images of carried items using the camera of the user terminal.

[1402] "Prohibited or restricted items" are items that are prohibited or restricted from being carried on board by the airline or the country of destination.

[1403] "Means for detection" refers to the function of analyzing the image data received by the server and identifying prohibited or restricted items.

[1404] "Means for notifying" refers to a function that sends the detection results to the user terminal and prompts the user to take appropriate action.

[1405] "Means for capturing audio and video data in real time" refers to the ability to record environmental sounds and videos in real time using the camera and microphone of the user terminal.

[1406] "Emotion engine" refers to an analysis function for recognizing the user's emotional state from audio and video data.

[1407] "Suspicious behavior and speech" refers to abnormal behavior or speech that is not in line with normal patterns of behavior and is deemed to pose a security risk.

[1408] "Security Officer" refers to the personnel who conduct inspections, monitoring, and respond as necessary.

[1409] "Means for analyzing the recognition results of the emotion engine by the server" refers to a function for the server to perform behavioral analysis based on data obtained from the emotion engine.

[1410] "Means of notifying security personnel in real time" refers to the function of immediately sending an alert to security personnel when abnormal behavior or speech is detected.

[1411] The present invention is a system centered around a user terminal, a server, an emotion engine, and the exchange of data between them. The main components of this system and their functions are specifically shown below.

[1412] Hardware and software used

[1413] User device: Smartphone or dedicated device

[1414] Server: Cloud Server

[1415] Emotion Engine: EmotionEngine (emotion recognition module)

[1416] Image processing software: OpenCV

[1417] Server communication library: requests

[1418] Customs declaration generation software: TaxDeclarationGenerator (automatic generation module)

[1419] Data processing and calculation

[1420] 1. Taking and analyzing images of personal belongings

[1421] When a user arrives at the airport, they use the camera on their device to take a picture of their belongings. The captured image is temporarily saved on the device. The image is then encoded using OpenCV, an image processing software, and uploaded to the server. The server receives the uploaded image data and uses AI technology to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item and notifies the user's device of the result.

[1422] 2. Travel information collation and notification

[1423] The server compares the user's passport information with the regulations of the destination country based on the image analysis results. If a prohibited or restricted item is detected, the server notifies the user of the results and prompts the user to take necessary measures. For example, if a prohibited item such as a "lighter" is detected, the server notifies the user that "lighters are not permitted on board. Please check them in as luggage."

[1424] 3. Automatic generation of customs declaration documents

[1425] The server automatically generates a customs declaration form based on the list of recognized items. This process includes determining whether the items are subject to declaration. The automatically generated customs declaration form is sent to the user's terminal, where the user can use it for customs procedures.

[1426] 4. Emotion Recognition and Suspicious Behavior Detection at Entry

[1427] During immigration inspections, the user's device captures audio and video data in real time and sends it to the server. The emotion engine recognizes the user's emotions from this data, and the server analyzes the recognition results. If suspicious behavior or speech is detected, the server immediately notifies security personnel. For example, if the user's history is normal but significant tension or anxiety is detected, the server will transmit that information to security personnel in real time.

[1428] 5. Personalized notifications and relaxation guide

[1429] Based on the emotion recognition results, the server provides personalized notifications and advice to the user. For example, if tension or anxiety is detected, a notification offering guidance and voice advice on how to relax will be sent to the user's device. This reduces the user's stress and provides a comfortable travel experience.

[1430] Examples of concrete examples and prompts

[1431] Example: Security Assistant App Usage Scenario

[1432] When a user arrives at the airport, they take a photo of their belongings and upload it before checking in. The image is analyzed and a notification is sent to them that the detected "lighter" is not permitted. Later, if the user becomes nervous during immigration, the emotion engine detects this and the server notifies the security officer.

[1433] Prompt Sentence Examples

[1434] Consider a scenario where a user takes and uploads a picture of their baggage at an airport. The image is analyzed to classify the items according to airline and destination country regulations, detect any illegal items, and then recognize the user's emotions in real time, notifying security if suspicious behavior is detected.

[1435] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1436] Step 1:

[1437] The user arrives at the airport and takes a picture of their belongings using the camera on the user terminal.

[1438] Input: Image of belongings

[1439] Output: Captured image file

[1440] Specific operation: Capture an image of the item with the user's device's camera app and save it as an image file on the device.

[1441] Step 2:

[1442] The captured image files are encoded using OpenCV and uploaded to the server.

[1443] Input: Image file

[1444] Output: Send image data to the server

[1445] Specific operation: Encodes an image using the OpenCV library on the user device, and sends the image data to the server as an HTTP POST request using the requests library.

[1446] Step 3:

[1447] The server analyzes the received image data and uses AI technology to recognize and classify the items.

[1448] Input: Encoded image data

[1449] Output: Analysis results (list of recognized items, detection results of prohibited and restricted items)

[1450] How it works: The server uses an AI model to analyze image data, automatically classify recognized items, and generate detection results for prohibited and restricted items.

[1451] Step 4:

[1452] The server notifies the user of the analysis results and prompts the user to take necessary measures.

[1453] Input: Analysis results (detection results of prohibited and restricted items)

[1454] Output: Notification message to the user's terminal

[1455] Specific operation: The server generates a notification message based on the detection result and sends it to the user's device. For example, if a prohibited item such as a "lighter" is detected, the server notifies the user by saying, "Lighters are prohibited from being carried on board. Please put them in checked baggage."

[1456] Step 5:

[1457] The server compares the analysis results with travel information to check whether the items being carried are in compliance with regulations.

[1458] Input: Analysis results, travel information (passport information)

[1459] Output: Whether or not there is a violation and a notification message

[1460] Specific operation: The server compares the travel information with the analysis results, and if any illegal items are found, sends a message to the user's device.

[1461] Step 6:

[1462] During immigration control, the user terminal captures audio and video data in real time and transmits it to the server.

[1463] Input: Real-time audio and video data

[1464] Output: Sending audio and video data to the server

[1465] Specific operation: Captures environmental sounds and videos in real time using the camera and microphone of the user device and sends them to the server.

[1466] Step 7:

[1467] The server analyzes the received audio and video data using an emotion engine to recognize the user's emotions.

[1468] Input: Real-time audio and video data

[1469] Output: Emotion recognition result

[1470] Specific operation: The server uses EmotionEngine to analyze audio and video data and recognize the user's emotional state.

[1471] Step 8:

[1472] The server analyzes the emotion recognition results and notifies security personnel if any suspicious behavior or speech is detected.

[1473] Input: Emotion recognition results

[1474] Output: A message to notify the security officer

[1475] Specific operation: The server analyzes the emotion recognition results, and if any suspicious behavior or speech is detected, it sends a message to the security officer in real time.

[1476] Step 9:

[1477] The server provides personalized notifications and advice to the user based on the emotion recognition results.

[1478] Input: Emotion recognition results

[1479] Output: A personalized notification message to the user

[1480] Specific operation: If the server detects tension or anxiety based on the emotion recognition results, it sends a notification message to the user's device containing relaxation guidance and advice.

[1481] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1482] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1483] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1484] [Fourth embodiment]

[1485] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1486] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1487] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1488] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1489] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1490] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1491] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1492] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1493] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1494] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1495] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1496] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1497] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1498] The embodiment of the present invention will be described with a focus on a user terminal, a server, and data exchange between them.

[1499] Advance preparation before departure and image recognition

[1500] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[1501] Device: The captured images are temporarily stored on the device, which then uploads them to the server.

[1502] Server: The server analyzes the received image data. During the analysis, AI technology is used to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item. The result of the determination is sent to the user's device.

[1503] Travel information collation and notification

[1504] Server: After obtaining the image analysis results, the server checks the user's passport information. Specifically, it compares the user's belongings with the destination country's regulations and airline regulations to see if they violate any rules. If prohibited or restricted items are detected, the server notifies the user.

[1505] Terminal: The user terminal displays the notification received from the server and prompts the user to take necessary measures, such as changing prohibited items into checked baggage.

[1506] Auto-generation of customs declaration documents

[1507] Server: The server automatically generates the necessary customs declaration forms based on the list of recognized items, including determining whether the items require declaration.

[1508] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed to the user on the terminal, allowing the user to follow the instructions and prepare for a smooth customs process.

[1509] Smuggling detection at the point of entry

[1510] Device: During immigration procedures, the user's device captures audio and video data and transmits it to a server in real time, allowing the user's actions and behavior during the procedure to be monitored.

[1511] Server: The server analyzes the received audio and video data to detect suspicious behavior or actions, such as when a user makes contradictory statements or exhibits specific non-verbal behavior.

[1512] Server: When suspicious activity is detected, the server notifies security personnel in real time, allowing for any necessary further investigations or responses to be taken promptly.

[1513] Specific examples

[1514] Example 1: Image recognition before departure

[1515] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[1516] Example 2: Automatic generation of customs declaration documents

[1517] When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. By presenting this form at the customs office, the user can smoothly proceed with customs clearance.

[1518] Example 3: Detecting contraband

[1519] If a user behaves suspiciously during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security, and additional measures are taken.

[1520] This system will reduce immigration issues and significantly improve the efficiency of security and customs inspections.

[1521] The processing flow will be explained below.

[1522] Step 1:

[1523] User: Takes pictures of their belongings using a smartphone or dedicated device. The user takes pictures of all items in turn and records the necessary information.

[1524] Step 2:

[1525] Device: Temporarily stores captured images. Once all items have been photographed, the image data will be ready to upload.

[1526] Step 3:

[1527] On the device: Upload the captured image to the server. Display a notification to the user when the upload is complete.

[1528] Step 4:

[1529] Server: Analyzes the received image data. Using AI technology, it recognizes and classifies the objects in each image.

[1530] Step 5:

[1531] Server: Compares the analysis results with the destination country's regulations and airline rules to identify prohibited or restricted items.

[1532] Step 6:

[1533] Server: If prohibited or restricted items are detected, it generates a list of them and creates notification data.

[1534] Step 7:

[1535] Terminal: Receives notification data sent from the server and displays it to the user. The user checks the displayed information and takes necessary measures.

[1536] Step 8:

[1537] Server: Based on the list of recognized items, a customs declaration form is automatically generated, listing the items that must be declared.

[1538] Step 9:

[1539] Server: Sends automatically generated customs declaration documents to the user's terminal.

[1540] Step 10:

[1541] Terminal: Displays the customs declaration documents sent to the user's terminal. The user checks the contents of the documents and prepares for customs procedures.

[1542] Step 11:

[1543] Terminal: During immigration procedures, audio and video data is captured and transmitted to a server in real time.

[1544] Step 12:

[1545] Server: Analyzes received audio and video data to detect suspicious behavior and speech, including non-verbal behavior and contradictory statements.

[1546] Step 13:

[1547] Server: If suspicious activity is detected, it notifies security personnel in real time and sends the analysis results as additional information.

[1548] Step 14:

[1549] Security personnel: Receive notifications from the server and promptly carry out any necessary additional checks or actions.

[1550] Example 1

[1551] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1552] Conventional immigration control systems require users to perform numerous manual tasks when handling carry-on baggage and customs declaration procedures, resulting in inefficiency and a high likelihood of errors. Furthermore, detecting smuggling during immigration inspections relies on human resources, making thorough monitoring difficult. Therefore, there is a need for a system that can improve efficiency and prevent fraud in each process of carry-on baggage management, customs declaration, and immigration inspection.

[1553] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1554] In this invention, the server includes means for taking images of carried items using a user terminal, means for temporarily saving the images taken by the terminal, means for uploading the images saved by the terminal to the server, means for analyzing the image data received by the server using AI technology to recognize and classify the items, means for determining whether the items fall under prohibited or restricted items, and means for transmitting the determination result to the user terminal. This automates the management of carried items, reduces the burden on users, and enables the prevention of fraudulent activities and the efficiency of immigration procedures.

[1555] A "user terminal" is an electronic device such as a mobile information terminal or a smartphone that is intended to be operated by a user.

[1556] "Personal effects" refers to all items that a user carries with them when traveling or moving.

[1557] The "means for taking an image" is a function for taking an image of a carried item using a camera function provided in the user terminal.

[1558] The "temporary storage means" is a function for storing captured images in a temporary storage device within the user terminal.

[1559] A "server" is a remote data processing device that receives and processes data sent from a user terminal.

[1560] "Uploading means" is a function for transmitting data from a user terminal to a server.

[1561] "Means for analyzing image data using AI technology" refers to a processing function that uses AI technology on a server to identify and classify objects contained in images.

[1562] "Means for recognizing and classifying items" refers to a function that identifies and classifies the name and type of an item from image data analyzed by AI.

[1563] "Means for determining whether a recognized item is a prohibited or restricted item" is a function that determines whether the recognized item violates the laws and regulations of the destination country.

[1564] The "means for transmitting the determination result to the user terminal and notifying the user" is a function for transmitting the determination result to the user terminal and displaying a notification to the user.

[1565] "Travel information" refers to all information related to a user's travel, such as passport information and airline ticket information.

[1566] "Means of matching" refers to the function of comparing the image analysis results with travel information to check for matches or violations.

[1567] The "means for automatically generating customs declaration documents" is a function that automatically creates the necessary customs declaration documents based on the analyzed carry-on item data.

[1568] The "means for displaying the automatically generated customs declaration document on the user terminal" is a function for transmitting the generated customs declaration document to the user terminal so that the user can view it on the terminal.

[1569] The present invention relates to a system for improving the efficiency of carry-on baggage management and customs declaration procedures at the time of immigration and for strengthening smuggling detection. Specific embodiments of the system are described below.

[1570] Preparation before departure and image recognition

[1571] User: Before leaving the country, the user takes pictures of their belongings using their smartphone or a dedicated device. This is done using the smartphone's camera function. For example, the user can take pictures of each item they put in their suitcase, obtaining image data of all of their belongings.

[1572] On the device: The captured images are saved in a temporary folder (local storage) on the user's device. They are then automatically uploaded from the device to the server. This upload process is carried out via a dedicated application on the device.

[1573] Server: The server analyzes the received image data using AI technology. Specifically, it uses image analysis libraries such as TensorFlow and OpenCV to recognize and classify objects. For example, objects in the image are recognized as "lighter," "bottle," "book," etc.

[1574] Server: Based on the analysis results, the server determines whether the item is a prohibited or restricted item. This involves checking it against a list of prohibited items stored on the server. The result of the determination is then sent to the user's device. The user is notified and can take the necessary measures accordingly.

[1575] Travel information collation and notification

[1576] Server: After obtaining the image analysis results, the server checks the user's passport information (electronic passport data) and travel information to ensure that the belongings comply with the regulations of the destination country and airline.

[1577] Server: If a prohibited or restricted item is detected, the server sends that information to the user's device and notifies the user of the appropriate action to take, such as "Lighters are not permitted on board. Please put them in checked baggage."

[1578] Auto-generation of customs declaration documents

[1579] Server: The server automatically generates the necessary customs forms based on the list of recognized items, including determining whether the items require declaration.

[1580] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed in a PDF viewer or dedicated application, allowing the user to follow the instructions and prepare for smooth customs procedures.

[1581] Smuggling detection at the point of entry

[1582] Terminal: During immigration control, the user's terminal captures audio and video data and transmits it to the server in real time using the terminal's built-in camera and microphone.

[1583] Server: The server analyzes the received audio and video data to detect suspicious behavior and speech. For example, it uses voice recognition and facial recognition systems to analyze the user's speech and non-verbal behavior (e.g., facial expressions and body movements).

[1584] Server: If any suspicious activity is detected, the server immediately notifies security personnel and prompts them to take any necessary further investigations or actions, thereby monitoring potential smuggling activity in real time.

[1585] Specific examples

[1586] Example 1: Before leaving the country, a user takes a picture of their baggage with their smartphone, and the device sends the picture to a server. If the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as checked baggage."

[1587] Example 2: When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. The user then presents the form at the customs office and the customs clearance process goes smoothly.

[1588] Example 3: If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security, and additional measures are taken.

[1589] This system will reduce immigration issues and significantly improve the efficiency of security and customs inspections.

[1590] Prompt Sentence Examples

[1591] Example: "How can I take a picture of my baggage before leaving the country and upload it to a server, so that AI can recognize and classify the items and notify the user if any prohibited or restricted items are detected?"

[1592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1593] Step 1:

[1594] User: The user uses a smartphone or dedicated device to take pictures of each item in their baggage. Specifically, the user launches a camera app and takes pictures of each item of baggage.

[1595] Input: The physical state of the baggage.

[1596] Output: Digital image data.

[1597] Step 2:

[1598] Device: The captured image is automatically saved to a temporary folder on the user's device.

[1599] Input: Images taken with a camera app on a smartphone or dedicated device.

[1600] Output: Image file saved in the device's local storage.

[1601] Step 3:

[1602] On the device: To upload the saved image to the server, send the image data via the API. Specifically, send the image data to the server using an HTTP request.

[1603] Input: An image file stored in local storage.

[1604] Output: Image data sent to the server.

[1605] Step 4:

[1606] Server: The server analyzes the received image data using AI technology, for example, using image analysis libraries such as TensorFlow or OpenCV, and applies a model to recognize objects in the image.

[1607] Input: Image data sent to the server.

[1608] Output: Identified item information as a result of analysis (e.g. lighter, plastic bottle, book, etc.).

[1609] Step 5:

[1610] Server: Based on the analysis results, it determines whether the item is a prohibited or restricted item, and in this process, it checks against the prohibited items list.

[1611] Input: Identified item information as a result of analysis.

[1612] Output: Judgment result (e.g. prohibited item, restricted item, permitted item).

[1613] Step 6:

[1614] Server: Sends the judgment result to the user device. Specifically, it generates JSON data in the form of a notification and sends it to the user device.

[1615] Input: The result of the test.

[1616] Output: Notification data sent to the user device.

[1617] Step 7:

[1618] Terminal: The user terminal analyzes the received notification data and displays it to the user. Specifically, it uses a pop-up notification or an alert dialog to alert the user.

[1619] Input: Notification data sent by the server.

[1620] Output: The notification displayed on the user's screen.

[1621] Step 8:

[1622] Server: Compares the user's passport and travel information with the image analysis results, specifically by accessing a database and comparing them with the destination country's regulations and airline regulations.

[1623] Input: Passport information, travel information, and image analysis results.

[1624] Output: Verification data as a result of the match (e.g., whether or not there is a violation).

[1625] Step 9:

[1626] Server: If prohibited or restricted items are detected based on the matching results, the server notifies the user of this information.

[1627] Input: Verification data as a match result.

[1628] Output: The alert data sent to the user terminal.

[1629] Step 10:

[1630] Terminal: The user's terminal analyzes the received warning data and notifies the user of countermeasures. For example, it displays specific instructions such as "Lighters are prohibited from being carried on board. Please put them in checked baggage."

[1631] Input: The alert data sent by the server.

[1632] Output: The instructions displayed on the user's screen.

[1633] Step 11:

[1634] Server: Automatically generates customs declaration documents based on the list of recognized carry-on items. Specifically, it automatically fills in the necessary information in a template and generates the documents in PDF format.

[1635] Input: A list of recognized items.

[1636] Output: Auto-generated customs declaration form (PDF).

[1637] Step 12:

[1638] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed for the user to view.

[1639] Input: Auto-generated customs declaration form (PDF).

[1640] Output: Customs declaration document displayed on user's terminal.

[1641] Step 13:

[1642] Terminal: During immigration control, the user terminal captures audio and video data and transmits it to the server in real time using the built-in camera and microphone.

[1643] Input: Audio and video data from immigration inspection.

[1644] Output: Audio and video data sent to the server.

[1645] Step 14:

[1646] Server: The server analyzes the received audio and video data and detects suspicious behavior or speech. Specifically, it uses voice recognition and facial recognition systems.

[1647] Input: Audio and video data sent to the server.

[1648] Output: Evaluation data of behavior and speech as the analysis result.

[1649] Step 15:

[1650] Server: If suspicious activity is detected, the server notifies security personnel in real time, facilitating further investigation and response.

[1651] Input: Evaluation data of behavior and speech as the analysis results.

[1652] Output: Alert data sent to security personnel.

[1653] This processing step reduces immigration issues and significantly improves the efficiency of security and customs inspections.

[1654] (Application example 1)

[1655] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1656] When traveling internationally, it is cumbersome to check whether your belongings comply with regulations before leaving the country, and it takes time to fill out customs declaration forms. Furthermore, there is a lack of means to quickly identify travelers exhibiting suspicious behavior during immigration inspections, making it difficult to ensure security. There is a need for a solution to these problems, making immigration procedures more efficient and improving security.

[1657] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1658] In this invention, the server includes means for taking images of carried items using a user terminal, means for analyzing the images taken by the server to detect prohibited or restricted items, means for notifying the user of the detection results, means for automatically generating customs declaration forms and displaying them on the user terminal, means for capturing audio and video data during immigration inspection and transmitting it to the server in real time, means for analyzing the audio and video data in the server to detect suspicious behavior, and means for notifying security personnel when suspicious behavior is detected. This enables quick confirmation of carried items before departure, automates the creation of customs declaration forms, and strengthens security inspections at the time of entry.

[1659] A "user terminal" is an electronic device that is carried and operated by a user, and is a device for taking images of carried items and receiving notifications.

[1660] A "server" is a computer system connected to a network that stores, analyzes, and processes data.

[1661] "Personal effects" are items that a user carries with them when traveling or going out.

[1662] The "means for taking an image" is a mechanism for taking a photo of a carried item using the camera function installed in the user terminal.

[1663] The "means for analyzing images" refers to algorithms or software for processing the image data received by the server and detecting specific items.

[1664] "Prohibited items" are items that are prohibited from being possessed or brought into the country under laws and regulations.

[1665] "Restricted items" are items whose possession or import is restricted based on permits or conditions.

[1666] "Means for notifying" refers to a technology for transmitting information from a server to a user terminal in real time and informing the user.

[1667] "Customs Declaration Form" means an official document used to declare possession of goods to customs.

[1668] "Automatic generation means" refers to a system that uses a server algorithm to automatically generate the necessary documents.

[1669] "Means for capturing audio and video data" refers to the process of acquiring audio and video using a microphone and camera installed on the user terminal.

[1670] "Means for transmitting in real time" refers to Internet communication technology for instantly transmitting data to a server.

[1671] "Means for detecting suspicious behavior" refers to technology that automatically detects suspicious behavior and speech based on audio and video data on the server.

[1672] "Means for notifying security personnel" refers to a system for sending warning information to security personnel when the server detects suspicious activity.

[1673] The present invention provides a system for improving the efficiency of baggage management before departure and security checks at the time of entry through data exchange between a user terminal and a server. Specific embodiments of the system are described below.

[1674] Baggage management before departure

[1675] 1. Taking pictures using a user device

[1676] A user takes a picture of their belongings with a user terminal such as a smartphone, and image data of the baggage is acquired using the camera of the smartphone.

[1677] 2. Sending image data

[1678] The captured image data is uploaded from the user's device to a server via an internet connection, and the data is sent in real time.

[1679] 3. Server Image Analysis

[1680] The server analyzes the received image data using AI technology. Specifically, it uses an object recognition algorithm to identify items in the image and detect prohibited or restricted items. For this, image analysis software such as OpenCV or TensorFlow can be used.

[1681] 4. Notification of test results

[1682] The analysis results are sent to the user's device, where the user can check whether or not there are any prohibited or restricted items. If the user accidentally carries a prohibited item, the device will notify the user in real time.

[1683] 5. Automatic generation of customs declaration documents

[1684] The server automatically generates customs declaration forms based on the analyzed list of carried items, and the generated forms are sent to the user's terminal, where the user can view and print them.

[1685] Security checks at entry

[1686] 1. Audio and Video Capture

[1687] During immigration inspection, the user's device (smartphone) captures audio and video data in real time and transmits it to a server using the smartphone's microphone and camera.

[1688] 2. Real-time data analysis

[1689] The server analyzes the received audio and video data, using natural language processing technology for audio analysis and object detection and motion analysis technology for video analysis, to detect suspicious behavior or contradictory statements in real time.

[1690] 3. Security Notice

[1691] If any suspicious activity is detected, the server immediately notifies security personnel, allowing any necessary additional testing or response to be carried out promptly.

[1692] Examples of specific examples and prompts

[1693] Example 1: Image recognition before departure

[1694] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[1695] Example 2: Security checks at entry

[1696] If a user behaves suspiciously during immigration control, the user's device captures the information and the server analyzes it in real time. If any unnatural behavior is detected, the server immediately notifies security personnel, who then take additional action.

[1697] Prompt for the generative AI model (luggage analysis example):

[1698] "Analyze the image to determine if it contains any illegal items. The image data is contained in the next argument, image_data."

[1699] The above embodiment is expected to improve the efficiency of immigration procedures and security.

[1700] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1701] Step 1:

[1702] A user takes a photo of their belongings with a user terminal such as a smartphone. The captured image data is acquired using the smartphone's camera. The input is the image data captured by the user's operation, and the output is the image data temporarily saved on the terminal.

[1703] Step 2:

[1704] The device uploads the captured image data to a server. The data is transmitted in real time via an internet connection. The input is the captured image data, and the output is the image data transmitted to the server.

[1705] Step 3:

[1706] The server analyzes the received image data using AI technology. Specifically, it uses object recognition algorithms (e.g., OpenCV or TensorFlow) to identify items in the image and detect prohibited or restricted items. The input is the image data sent to the server, and the output is the analysis results.

[1707] Step 4:

[1708] The server sends a notification to the user's device based on the analysis results. The notification includes details of the prohibited or restricted items detected. The input is the result data from the AI ​​analysis, and the output is a notification message sent to the user's device.

[1709] Step 5:

[1710] The server automatically generates customs declaration forms based on the analyzed list of carried items, including determining which items should be declared. The input is the list of items based on the analysis results, and the output is the generated customs declaration form.

[1711] Step 6:

[1712] The server sends the automatically generated customs declaration form to the user's terminal, where the user can check and print it. The input is the generated customs declaration form, and the output is the declaration form sent to the user's terminal.

[1713] Step 7:

[1714] During immigration control, the user's device (smartphone) captures audio and video data and transmits it to the server in real time. The data is acquired using the smartphone's microphone and camera. The input is the audio and video data captured at the immigration control site, and the output is the audio and video data transmitted to the server.

[1715] Step 8:

[1716] The server analyzes the received audio and video data. It uses natural language processing technology for audio analysis and object detection and motion analysis technology for video analysis. The input is the captured audio and video data, and the output is behavioral judgment data as the analysis result.

[1717] Step 9:

[1718] Based on the analysis results, the server determines whether there is any suspicious behavior or contradictory statements, and notifies security personnel if any suspicious behavior is detected. The input is behavior determination data, and the output is a notification message to the security personnel.

[1719] These steps ensure efficient and effective baggage checks before departure, automatic generation of customs declaration forms, and security checks upon entry.

[1720] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1721] In the present invention, a specific embodiment for implementing a system centered on a user terminal, a server, an emotion engine, and data exchange between these will be described.

[1722] Advance preparation before departure and image recognition

[1723] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[1724] Device: The captured images are temporarily stored on the device, which then uploads them to the server.

[1725] Server: The server analyzes the received image data. It uses AI technology to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item. The result of the determination is sent to the user's device.

[1726] Travel information collation and notification

[1727] Server: After obtaining the image analysis results, the server checks the user's passport information. Specifically, it checks the user's belongings against the regulations of the destination country and the airline to see if they violate any rules. If any prohibited or restricted items are detected, the server notifies the user.

[1728] Terminal: The user terminal displays the notification received from the server and prompts the user to take necessary measures, such as changing prohibited items into checked baggage.

[1729] Auto-generation of customs declaration documents

[1730] Server: The server automatically generates the necessary customs declaration forms based on the list of recognized items, including determining whether the items require declaration.

[1731] Terminal: The automatically generated customs declaration form is sent to the user's terminal and displayed to the user on the terminal, allowing the user to follow the instructions and prepare for a smooth customs process.

[1732] Smuggling detection and emotion recognition at immigration

[1733] Terminal: During immigration procedures, the user's terminal captures audio and video data and transmits it to the server in real time. The emotion engine also recognizes the user's emotions from the audio and video data.

[1734] Server: The server analyzes the received audio and video data and the emotion recognition results obtained from the emotion engine to detect suspicious behavior or speech. For example, it determines if the user makes contradictory statements or exhibits specific non-verbal behavior.

[1735] Server: When suspicious behavior or negative sentiment is detected, the server notifies security personnel in real time, allowing for any necessary further investigations or responses to be taken promptly.

[1736] Emotion-based personalization

[1737] Server: Based on the emotion recognition results obtained from the emotion engine, the server provides personalized notifications and advice to the user. For example, if tension or anxiety is detected, the server provides guidance and advice on how to relax.

[1738] Device: The user's device receives this personalized notification and provides the user with appropriate information to support a comfortable travel experience.

[1739] Specific examples

[1740] Example 1: Image recognition before departure

[1741] For example, a user takes a picture of their baggage with their smartphone before leaving the country. The device sends the picture to a server, and if the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as baggage."

[1742] Example 2: Automatic generation of customs declaration documents

[1743] When a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. By presenting this form at the customs office, the user can smoothly proceed with customs clearance.

[1744] Example 3: Contraband detection and emotion recognition

[1745] If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If unnatural behavior is detected, the server immediately notifies security, and additional measures are taken. If the emotion engine detects tension or anxiety in the user, the server sends a notification offering advice to the user on how to relax.

[1746] The system will reduce immigration issues, significantly improve the efficiency of security and customs inspections, and provide appropriate support based on the user's emotional state, ensuring a comfortable travel experience.

[1747] The processing flow will be explained below.

[1748] Step 1:

[1749] User: Before leaving the country, the user takes a picture of their belongings using a smartphone or a dedicated device, which records detailed information about their luggage.

[1750] Step 2:

[1751] Device: Images are temporarily stored on the device. Once all items have been photographed, the image data will be ready to upload.

[1752] Step 3:

[1753] On the device: Upload the captured image to the server. Display a notification to the user when the upload is complete.

[1754] Step 4:

[1755] Server: Analyzes the received image data. Using AI technology, it recognizes and classifies the objects in each image.

[1756] Step 5:

[1757] Server: Compares the analysis results with the destination country's regulations and airline rules to identify prohibited or restricted items.

[1758] Step 6:

[1759] Server: If prohibited or restricted items are detected, it generates a list of them and creates notification data.

[1760] Step 7:

[1761] Terminal: Receives notification data sent from the server and displays it to the user. The user checks the displayed information and takes necessary measures.

[1762] Step 8:

[1763] Server: Based on the list of recognized items, a customs declaration form is automatically generated, listing the items that must be declared.

[1764] Step 9:

[1765] Server: Sends automatically generated customs declaration documents to the user's terminal.

[1766] Step 10:

[1767] Terminal: Displays the customs declaration documents sent to the user's terminal. The user checks the contents of the documents and prepares for customs procedures.

[1768] Step 11:

[1769] Terminal: During immigration procedures, audio and video data is captured and sent to the server in real time. An emotion engine also runs to recognize the user's emotions.

[1770] Step 12:

[1771] Server: Analyzes the received audio and video data and emotion recognition results obtained from the emotion engine to detect suspicious behavior or speech. For example, it determines if the user makes contradictory statements or engages in specific non-verbal behavior.

[1772] Step 13:

[1773] Server: When suspicious behavior or negative sentiment is detected, security personnel are notified in real time, allowing for any necessary further investigation or response to be taken promptly.

[1774] Step 14:

[1775] Server: Provides personalized notifications and advice to users based on the emotion recognition results obtained from the emotion engine. For example, if tension or anxiety is detected, it sends guidance and advice on how to relax.

[1776] Step 15:

[1777] Device: The user's device receives the personalized notification and displays appropriate information to the user, who can then check the notification and take action to relax.

[1778] Example 2

[1779] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1780] In recent years, with the increase in cross-border travel, safety and efficiency at immigration are required, but current systems have difficulty in detecting the bringing in of prohibited or restricted items in advance. Another issue is that customs declaration procedures and security checks at the point of entry require time and effort from users. This reduces convenience for travelers and the efficiency of security and customs inspections.

[1781] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for taking an image of the carried items using the user terminal, a means for temporarily saving the image taken by the terminal and uploading it to the server, a means for analyzing the image data received by the server and classifying the identified items, a means for determining whether the identified items fall under prohibited or restricted items, and a means for notifying the user terminal of the determination result. This enables the user to quickly check the carried items before departure and take necessary measures. In addition, customs declaration procedures are simplified and security checks at the time of entry are made more efficient.

[1782] A "user terminal" is an electronic device that a user uses to take pictures of their belongings and upload the data to a server.

[1783] The "means for taking an image" is a function that allows a user to capture an image of an item carried by the user using the user terminal.

[1784] "Terminal" refers to a user terminal or a device that has the function of temporarily storing captured images and uploading them to a server.

[1785] The "means for temporarily saving images" is a function for temporarily saving captured images on the terminal.

[1786] The "server" is a central management system that analyzes the received image data, classifies the identified items, and notifies the results to the user terminal.

[1787] The "means for analyzing image data" refers to algorithms or software used by the server to analyze the image data received.

[1788] The "means for classifying identified items" is a function by which the server identifies items from image data and classifies them into categories.

[1789] The "means of determining whether an item is a prohibited or restricted item" is a function that uses analyzed data to check whether the identified item violates the regulations of the destination or airline.

[1790] The "means for notifying the user terminal of the determination result" is a function by which the server sends a message to the user terminal to notify the user of the determination result.

[1791] "Travel information" is data such as the user's passport information and travel destination information.

[1792] The "means for automatically generating customs declaration documents" is a function that allows the server to automatically generate customs declaration documents based on the list of carried items.

[1793] The present invention provides a system for travelers to efficiently and safely complete immigration procedures. The system includes a user terminal, a server, and a network for exchanging data between them.

[1794] Hardware and software used

[1795] User device: A device such as a smartphone or tablet that takes pictures of personal items and uploads the data to a server. This device must have a camera and internet connection.

[1796] Server: The server is a central management system that receives and analyzes image data. Deep learning frameworks such as "TensorFlow" and "YOLO" are used for analysis. MySQL and PostgreSQL are used as databases.

[1797] Network: Data exchange requires an internet connection and uses the HTTP or HTTPS protocol.

[1798] Specific system processing

[1799] Preparation before departure and image recognition

[1800] User: Before leaving the country, the user uses their smartphone to take pictures of their belongings. For example, they take pictures of the items in their suitcase one by one with the smartphone camera.

[1801] On the device: The captured image is temporarily stored in the device's storage, and then the device uploads the image to the server using the HTTP protocol.

[1802] Server: The server analyzes the received image data using TensorFlow and YOLO to automatically identify and classify the items. For example, it detects a "lighter" or "laptop" in the image. It compares the item with rules stored in a database to determine whether it is a prohibited or restricted item. The result of the determination is notified to the user's device.

[1803] Travel information collation and notification

[1804] Server: After receiving the image analysis results, the server compares the passport information with the regulations of the destination. For example, it can obtain the latest regulations of the destination through API and check whether the belongings violate the regulations. If any prohibited or restricted items are detected, the server will notify the user.

[1805] Device: The user device displays the notification received from the server, for example, a pop-up with specific instructions such as "Lighters are not allowed on board. Please put them in checked luggage."

[1806] Auto-generation of customs declaration documents

[1807] Server: The server automatically generates customs declaration forms based on the list of carried items. For example, it creates the declaration forms using a template engine such as "Jinja2." The generated forms are then sent to the user's device.

[1808] Terminal: The user terminal receives the automatically generated customs declaration form and displays it to the user, who then goes through customs procedures accordingly.

[1809] Smuggling detection and emotion recognition at immigration

[1810] Terminal: During immigration procedures, the user's terminal uses a camera and microphone to capture audio and video data, which is then transmitted to the server in real time. The emotion engine then recognizes the user's emotions from the audio and video data.

[1811] Server: The server performs analysis using emotion recognition engines such as Microsoft Azure Cognitive Services and IBM Watson. For example, it can detect user tension or anxiety. If suspicious behavior or negative emotions are detected, the server notifies security personnel in real time.

[1812] Emotion-based personalization

[1813] Server: The server provides personalized notifications and advice to users based on the emotion recognition results. For example, if tension is detected, the server sends a notification such as "Take a deep breath to relax."

[1814] Device: The user device receives this personalized notification and displays appropriate information to the user.

[1815] Examples of specific examples and prompts

[1816] Example 1: Before leaving the country, a user takes a picture of their baggage with their smartphone, and the device sends the picture to a server. If the server detects that a "lighter" is a prohibited item, the device notifies the user, "Lighters are prohibited from being carried on board. Please check them in as checked baggage."

[1817] Example 2: If a user carries expensive jewelry, the server determines that the jewelry must be declared and automatically generates a customs declaration form with details of the jewelry. The user then presents the form to the customs officer.

[1818] Example 3: If a user displays suspicious behavior during immigration control, the user's device captures the information and the server analyzes it in real time. If unnatural behavior is detected, the server immediately notifies security and additional measures are taken. If the emotion engine detects tension or anxiety in the user, the server sends a notification offering advice to the user on how to relax.

[1819] Example prompt sentence:

[1820] "Please take pictures of your belongings before you leave the country to make sure they do not contain any prohibited items."

[1821] "Auto-generate customs declaration documents that comply with the regulations of your destination."

[1822] "Detect smuggling and suspicious behavior in real time at the point of entry and provide security notifications based on emotion recognition results."

[1823] The system will reduce immigration issues, significantly improve the efficiency of security and customs procedures, and provide appropriate support based on the user's emotional state, ensuring a comfortable travel experience.

[1824] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1825] Step 1:

[1826] User: Before leaving the country, the user uses their smartphone to take a picture of their belongings.

[1827] Input: Image of belongings

[1828] Output: Image file saved in smartphone storage

[1829] Specific operation: The camera app on the smartphone is launched and each item is photographed individually. The photographed images are saved to the smartphone's internal storage or SD card.

[1830] Step 2:

[1831] On your device: Upload the saved image to the server.

[1832] Input: Image file saved in smartphone storage

[1833] Output: Image data sent to the server

[1834] What it does: It uses a smartphone app to send image data to a server via the HTTPS protocol, and displays a progress bar until all images taken by the user have been uploaded.

[1835] Step 3:

[1836] Server: Analyzes the received image data and identifies the item.

[1837] Input: Image data uploaded to the server

[1838] Output: List of identified items

[1839] How it works: The server runs deep learning models using TensorFlow and YOLO to identify the objects in each image, for example, determining whether the image contains a "lighter" or a "laptop."

[1840] Step 4:

[1841] Server: Determines whether the identified item is a prohibited or restricted item.

[1842] Input: List of identified items

[1843] Output: List of prohibited and restricted items

[1844] Specific operation: The system checks the regulations of the destination country stored in the database to determine whether the identified item is a prohibited or restricted item. For example, a "lighter" is detected as a prohibited item on board.

[1845] Step 5:

[1846] Server: Notifies the user of the result.

[1847] Input: List of prohibited and restricted items

[1848] Output: Notification message to the user's terminal

[1849] Specific operation: The server sends a push notification to the user's smartphone. For example, a message such as "Lighters are not permitted on board. Please check them in." is displayed.

[1850] Step 6:

[1851] Server: Automatically generate customs declaration documents.

[1852] Input: List of belongings

[1853] Output: Auto-generated customs declaration form

[1854] What happens: The server uses a template engine (e.g., Jinja2) to generate a customs declaration form from the list of items carried. The documents include details of the items that need to be declared.

[1855] Step 7:

[1856] Server: Sends automatically generated customs declaration documents to the user's terminal.

[1857] Input: Auto-generated customs declaration form

[1858] Output: Customs declaration document displayed on user's terminal

[1859] Specific operation: The server generates a customs declaration form, which is sent to the user's smartphone and displayed on the device. The user can save it as a PDF or present it directly to customs.

[1860] Step 8:

[1861] Device: During immigration procedures, a camera and microphone are used to capture audio and video data and transmit it to a server.

[1862] Input: Audio and video data

[1863] Output: Audio and video data sent to the server

[1864] Specific operation: During immigration procedures, the smartphone's camera and microphone are activated, and real-time audio and video recordings are sent to the server via the HTTPS protocol.

[1865] Step 9:

[1866] Server: Analyzes audio and video data and recognizes emotions.

[1867] Input: Audio and video data sent to the server

[1868] Output: Emotion recognition result

[1869] Specific operation: The server uses Microsoft Azure Cognitive Services and IBM Watson to analyze emotions from voice and recognize facial expressions from video to determine the user's emotional state.

[1870] Step 10:

[1871] Server: Notifies security personnel if suspicious behavior or negative sentiment is detected.

[1872] Input: Emotion recognition results

[1873] Output: Notification alert to security personnel

[1874] What it does: When abnormal behavior or negative sentiment is detected, the server sends real-time notifications to security personnel using a security alert system (e.g., PagerDuty).

[1875] Step 11:

[1876] Server: Provides personalized notifications and advice to users based on emotion recognition results.

[1877] Input: Emotion recognition results

[1878] Output: A personalized notification message

[1879] Specific operation: When a user feels nervous or anxious, the server generates an appropriate message and sends it to the user's smartphone, including advice such as "Take a deep breath to relax."

[1880] (Application example 2)

[1881] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1882] Immigration procedures, such as baggage inspections and customs procedures, are cumbersome and time-consuming, making it easy for fraud and violations to be overlooked. Furthermore, there is an insufficient system for recognizing suspicious passenger behavior and emotional states in real time and responding immediately, increasing security risks. These issues increase the burden on both travelers and inspectors, and create challenges for the overall travel experience.

[1883] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1884] In this invention, the server includes means for taking images of carried items using a user terminal, means for analyzing the taken images by the server and detecting prohibited or restricted items, means for notifying the user of the detection results, means for capturing audio and video data in real time and recognizing emotions using an emotion engine, and means for analyzing the recognition results of the emotion engine by the server to detect suspicious actions or behavior and notify security personnel in real time. This enables immigration procedures to be carried out quickly and accurately, and enables early detection and response to violations and fraudulent acts.

[1885] A "user terminal" is a device for taking pictures of personal items and capturing audio and video data.

[1886] A "server" is a central system that receives, analyzes, and processes data sent from user terminals.

[1887] "Personal effects" refers to all items that a user carries with them when entering or leaving a country.

[1888] "Means for taking images" refers to the function of capturing images of carried items using the camera of the user terminal.

[1889] "Prohibited or restricted items" are items that are prohibited or restricted from being carried on board by the airline or the country of destination.

[1890] "Means for detection" refers to the function of analyzing the image data received by the server and identifying prohibited or restricted items.

[1891] "Means for notifying" refers to a function that sends the detection results to the user terminal and prompts the user to take appropriate action.

[1892] "Means for capturing audio and video data in real time" refers to the ability to record environmental sounds and videos in real time using the camera and microphone of the user terminal.

[1893] "Emotion engine" refers to an analysis function for recognizing the user's emotional state from audio and video data.

[1894] "Suspicious behavior and speech" refers to abnormal behavior or speech that is not in line with normal patterns of behavior and is deemed to pose a security risk.

[1895] "Security Officer" refers to the personnel who conduct inspections, monitoring, and respond as necessary.

[1896] "Means for analyzing the recognition results of the emotion engine by the server" refers to a function for the server to perform behavioral analysis based on data obtained from the emotion engine.

[1897] "Means of notifying security personnel in real time" refers to the function of immediately sending an alert to security personnel when abnormal behavior or speech is detected.

[1898] The present invention is a system centered around a user terminal, a server, an emotion engine, and the exchange of data between them. The main components of this system and their functions are specifically shown below.

[1899] Hardware and software used

[1900] User device: Smartphone or dedicated device

[1901] Server: Cloud Server

[1902] Emotion Engine: EmotionEngine (emotion recognition module)

[1903] Image processing software: OpenCV

[1904] Server communication library: requests

[1905] Customs declaration generation software: TaxDeclarationGenerator (automatic generation module)

[1906] Data processing and calculation

[1907] 1. Taking and analyzing images of personal belongings

[1908] When a user arrives at the airport, they use the camera on their device to take a picture of their belongings. The captured image is temporarily saved on the device. The image is then encoded using OpenCV, an image processing software, and uploaded to the server. The server receives the uploaded image data and uses AI technology to recognize and classify the items. If necessary, it determines whether the photographed item is a prohibited or restricted item and notifies the user's device of the result.

[1909] 2. Travel information collation and notification

[1910] The server compares the user's passport information with the regulations of the destination country based on the image analysis results. If a prohibited or restricted item is detected, the server notifies the user of the results and prompts the user to take necessary measures. For example, if a prohibited item such as a "lighter" is detected, the server notifies the user that "lighters are not permitted on board. Please check them in as luggage."

[1911] 3. Automatic generation of customs declaration documents

[1912] The server automatically generates a customs declaration form based on the list of recognized items. This process includes determining whether the items are subject to declaration. The automatically generated customs declaration form is sent to the user's terminal, where the user can use it for customs procedures.

[1913] 4. Emotion Recognition and Suspicious Behavior Detection at Entry

[1914] During immigration inspections, the user's device captures audio and video data in real time and sends it to the server. The emotion engine recognizes the user's emotions from this data, and the server analyzes the recognition results. If suspicious behavior or speech is detected, the server immediately notifies security personnel. For example, if the user's history is normal but significant tension or anxiety is detected, the server will transmit that information to security personnel in real time.

[1915] 5. Personalized notifications and relaxation guide

[1916] Based on the emotion recognition results, the server provides personalized notifications and advice to the user. For example, if tension or anxiety is detected, a notification offering guidance and voice advice on how to relax will be sent to the user's device. This reduces the user's stress and provides a comfortable travel experience.

[1917] Examples of concrete examples and prompts

[1918] Example: Security Assistant App Usage Scenario

[1919] When a user arrives at the airport, they take a photo of their belongings and upload it before checking in. The image is analyzed and a notification is sent to them that the detected "lighter" is not permitted. Later, if the user becomes nervous during immigration, the emotion engine detects this and the server notifies the security officer.

[1920] Prompt Sentence Examples

[1921] Consider a scenario where a user takes and uploads a picture of their baggage at an airport. The image is analyzed to classify the items according to airline and destination country regulations, detect any illegal items, and then recognize the user's emotions in real time, notifying security if suspicious behavior is detected.

[1922] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1923] Step 1:

[1924] The user arrives at the airport and takes a picture of their belongings using the camera on the user terminal.

[1925] Input: Image of belongings

[1926] Output: Captured image file

[1927] Specific operation: Capture an image of the item with the user's device's camera app and save it as an image file on the device.

[1928] Step 2:

[1929] The captured image files are encoded using OpenCV and uploaded to the server.

[1930] Input: Image file

[1931] Output: Send image data to the server

[1932] Specific operation: Encodes an image using the OpenCV library on the user device, and sends the image data to the server as an HTTP POST request using the requests library.

[1933] Step 3:

[1934] The server analyzes the received image data and uses AI technology to recognize and classify the items.

[1935] Input: Encoded image data

[1936] Output: Analysis results (list of recognized items, detection results of prohibited and restricted items)

[1937] How it works: The server uses an AI model to analyze image data, automatically classify recognized items, and generate detection results for prohibited and restricted items.

[1938] Step 4:

[1939] The server notifies the user of the analysis results and prompts the user to take necessary measures.

[1940] Input: Analysis results (detection results of prohibited and restricted items)

[1941] Output: Notification message to the user's terminal

[1942] Specific operation: The server generates a notification message based on the detection result and sends it to the user's device. For example, if a prohibited item such as a "lighter" is detected, the server notifies the user by saying, "Lighters are prohibited from being carried on board. Please put them in checked baggage."

[1943] Step 5:

[1944] The server compares the analysis results with travel information to check whether the items being carried are in compliance with regulations.

[1945] Input: Analysis results, travel information (passport information)

[1946] Output: Whether or not there is a violation and a notification message

[1947] Specific operation: The server compares the travel information with the analysis results, and if any illegal items are found, sends a message to the user's device.

[1948] Step 6:

[1949] During immigration control, the user terminal captures audio and video data in real time and transmits it to the server.

[1950] Input: Real-time audio and video data

[1951] Output: Sending audio and video data to the server

[1952] Specific operation: Captures environmental sounds and videos in real time using the camera and microphone of the user device and sends them to the server.

[1953] Step 7:

[1954] The server analyzes the received audio and video data using an emotion engine to recognize the user's emotions.

[1955] Input: Real-time audio and video data

[1956] Output: Emotion recognition result

[1957] Specific operation: The server uses EmotionEngine to analyze audio and video data and recognize the user's emotional state.

[1958] Step 8:

[1959] The server analyzes the emotion recognition results and notifies security personnel if any suspicious behavior or speech is detected.

[1960] Input: Emotion recognition results

[1961] Output: A message to notify the security officer

[1962] Specific operation: The server analyzes the emotion recognition results, and if any suspicious behavior or speech is detected, it sends a message to the security officer in real time.

[1963] Step 9:

[1964] The server provides personalized notifications and advice to the user based on the emotion recognition results.

[1965] Input: Emotion recognition results

[1966] Output: A personalized notification message to the user

[1967] Specific operation: If the server detects tension or anxiety based on the emotion recognition results, it sends a notification message to the user's device containing relaxation guidance and advice.

[1968] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1969] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1970] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1971] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1972] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1973] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1974] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1975] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1976] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1977] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural networ...

Claims

1. means for taking an image of the carried item by a user terminal; A means for analyzing the captured image by the server and detecting prohibited or restricted items; The system includes a means for notifying a user of the detection results.

2. 2. The system according to claim 1, further comprising means for verifying the results of the analysis by the server against travel information linked to the passport, and confirming the prohibited or restricted items detected.

3. 2. The system according to claim 1, further comprising means for automatically generating a customs declaration form in the server based on the list of carried items and displaying the form on the user terminal.

4. means for capturing audio and video data by a user terminal and transmitting the data to a server; 2. The system according to claim 1, further comprising means for detecting suspicious behavior or speech through analysis of received data by the server and notifying the user of the detected behavior or speech.

Images