System

Abstract

A system is provided.SOLUTION: A system includes means for transmitting image data acquired by a camera mounted on an autonomous mobile apparatus, means for generating a prompt based on the image data, means for transmitting the prompt to a generative artificial intelligence model and receiving feedback regarding a no-go area, means for analyzing the feedback and mapping the no-go area, and means for managing information regarding the no-go area based on a retention period.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current autonomous mobile devices mainly use sensors such as LiDAR to detect obstacles, but they have limitations in their ability to recognize characters and detect thin objects (e.g., ropes and tape). For this reason, facility managers who operate robots must expend a great deal of effort and resources to temporarily restrict access to certain areas, such as during events. The objective of this invention is to provide a system that reduces this effort and enables autonomous mobile devices to autonomously determine restricted areas by analyzing camera images and efficiently take evasive action. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides a system including: means for transmitting image data acquired by a camera mounted on an autonomous mobile device; means for generating the image data as a prompt; means for transmitting the prompt to a generative artificial intelligence model and receiving feedback regarding no-entry areas; means for analyzing the feedback and mapping the no-entry areas; and means for managing information regarding the no-entry areas based on a retention period. The system further includes means for linking 2D image data and 3D point cloud data in the mapping to calculate spatial coordinates, and means for the autonomous mobile device to select an alternative route to a destination based on the information regarding the no-entry areas.

[0006] An "autonomous mobile device" is a device that uses artificial intelligence and sensor technology to recognize its own location and move to its destination while independently avoiding obstacles.

[0007] A "camera" is a device that converts light into an electrical signal and outputs it as image data.

[0008] "Image data" is visual information captured by an imaging device such as a camera, expressed in digital form.

[0009] A "prompt" is text data containing a specific question or command that is input into a generative AI model.

[0010] A "generative artificial intelligence model" is an artificial intelligence program that learns from large amounts of data and generates appropriate responses and predictions based on the input data.

[0011] "Feedback" refers to the response or instructions returned by a generative AI model in response to input data.

[0012] A "no-entry area" is an area where entry of autonomous mobile devices is restricted for safety reasons or for specific purposes.

[0013] "Mapping" is the process of illustrating spatial structures and spatial relationships based on geographical information and location data.

[0014] A "retention period" is the length of time that data or information is considered valid.

[0015] "2D image data" is digital data of visual information expressed on a two-dimensional plane.

[0016] "3D point cloud data" is digital data that indicates the position of each point in space using three-dimensional coordinates.

[0017] "Spatial coordinates" are numerical values ​​used to represent a position in three-dimensional space, and are usually expressed as x, y, and z values.

[0018] An "alternate route" is a newly planned route to reach the destination while avoiding the restricted area. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention relates to a system for an autonomous mobile device to autonomously determine a no-entry area and take action to avoid it. The elements required to implement the present invention and their operations will be described below in detail.

[0041] Server Operation

[0042] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. The specific operation of the server is described below.

[0043] 1. Receiving camera images

[0044] The server receives camera images periodically transmitted from the autonomous mobile device, and the received image data is temporarily stored in a buffer.

[0045] 2. Prompt Generation

[0046] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[0047] 3. Send to generative AI model

[0048] The server sends the generated prompt to the artificial intelligence model and waits for a response, which includes information about the no-entry area.

[0049] 4. Feedback Analysis

[0050] The feedback from the generative artificial intelligence model is analyzed to determine whether the area is a no-entry area.

[0051] 5. Mapping No-Go Areas

[0052] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. A depth camera is used to link 3D point cloud data with 2D images to perform detailed mapping.

[0053] 6. Information Retention Period Management

[0054] Information about restricted areas is retained for a set period based on feedback from the generative AI model, and is then deleted or updated from the database after the period expires.

[0055] Terminal (autonomous mobile device) operation

[0056] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. The specific operation of the autonomous mobile device is described below.

[0057] 1. Camera image capture

[0058] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[0059] 2. Sending images

[0060] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[0061] 3. Receiving Instructions

[0062] Receive feedback from the server and obtain information about restricted areas.

[0063] 4. Taking evasive action

[0064] Based on the information on the restricted area, the autonomous mobile device recalculates an alternative route to the destination and avoids the restricted area. If it is unable to plot a route to the destination, it notifies the administrator.

[0065] User (facility administrator) actions

[0066] The user manages and monitors the system. The specific actions of the user are described below.

[0067] 1. System Monitoring

[0068] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0069] 2. Change settings

[0070] Users can enter settings to prohibit entry to specific areas from the dashboard and send them to the server. Information on the designated area and period is sent to the server and reflected throughout the system.

[0071] 3. Check operation

[0072] The user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route and operation schedule.

[0073] Specific examples

[0074] The present invention will be described below with reference to specific examples.

[0075] Server example

[0076] 1. Receiving camera images

[0077] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[0078] 2. Prompt Generation

[0079] Generate a prompt containing the received image and format it in JSON format.

[0080] json

[0081] {

[0082] "image": " / tmp / captured_image.jpg",

[0083] "question": "Is this area off-limits?"

[0084] }

[0085] 3. Sending to the generative AI model and receiving feedback

[0086] This prompt is sent to the API endpoint of the generative AI model, and feedback is received saying, "This area is off-limits."

[0087] 4. Mapping No-Go Areas

[0088] The location of the no entry sign is converted into 3D coordinates and the map data of the autonomous mobile device is updated.

[0089] 5. Information Retention Period Management

[0090] The retention period information from the generative AI model, "This information is valid for one day," is registered in the database, and a deletion task is scheduled for one day later.

[0091] Examples of autonomous mobile devices

[0092] 1. Camera image capture

[0093] The autonomous mobile device uses a camera module to capture images within the facility.

[0094] 2. Sending images

[0095] Send the image data to the server using an HTTP POST request.

[0096] 3. Receiving Instructions

[0097] Receive feedback from the server that the area is off-limits and analyze the content.

[0098] 4. Taking evasive action

[0099] Based on information about restricted areas, a new route is calculated and travel to the destination is continued.

[0100] User Examples

[0101] 1. System Monitoring

[0102] Log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[0103] 2. Change settings

[0104] Settings are made to temporarily prohibit entry to specific areas and sent to the server.

[0105] 3. Check operation

[0106] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[0107] The processing flow will be explained below.

[0108] Server Processing Steps

[0109] Step 1:

[0110] The server waits for and receives camera images sent by the autonomous mobile device via HTTP requests, and temporarily stores the received image data in a buffer.

[0111] Step 2:

[0112] The server passes the received camera image to the prompt generation module, which generates a prompt containing the image and the text "Is this area off-limits?". The prompt is then formatted into JSON format to be sent to the generative AI model.

[0113] Step 3:

[0114] The server sends the formatted prompt via an HTTPS request to the API endpoint of the generative AI model and waits for feedback from the generative AI model.

[0115] Step 4:

[0116] The server receives feedback from the generative AI model in JSON format, including information such as "no entry areas," and analyzes the content to determine whether the area is a no entry area.

[0117] Step 5:

[0118] The server references spatial data linked to the camera's viewpoint to obtain coordinate information for the restricted area based on the feedback. It uses a depth camera or stereo method to link 3D data with 2D images and calculates detailed spatial coordinates.

[0119] Step 6:

[0120] The server adds the calculated spatial coordinates to the robot's map data and maps it as a new no-entry area.

[0121] Step 7:

[0122] The server stores the retention period information provided in the feedback of the generative AI model in a database, manages the retention period of the no-entry area information, and schedules a task to update the information when the retention period expires.

[0123] Processing steps of terminal (autonomous mobile device)

[0124] Step 1:

[0125] The autonomous mobile device captures images from the camera module at regular intervals and temporarily stores them in memory.

[0126] Step 2:

[0127] The autonomous mobile device sends the saved image to a specified server endpoint via an HTTP POST request. It also has a retry mechanism until the transmission is successful, ensuring communication stability.

[0128] Step 3:

[0129] The autonomous mobile device receives feedback from the server, analyzes the content, and obtains information about restricted areas.

[0130] Step 4:

[0131] The autonomous mobile device uses the received coordinate information of the no-entry area to autonomously calculate a new route and avoid the no-entry area.

[0132] Step 5:

[0133] The autonomous mobile device instructs the navigation module to find an avoidance route and continues moving to the destination. If the device cannot find a route to the destination, it notifies the administrator.

[0134] User (facility manager) processing steps

[0135] Step 1:

[0136] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0137] Step 2:

[0138] Users can set specific areas to be off-limits as needed by entering the area's coordinates and the off-limits period into the form on the management screen and pressing the "Set" button to send the information to the server.

[0139] Step 3:

[0140] The user checks whether the autonomous mobile device is properly avoiding restricted areas and adjusts the robot's route and operation schedule as necessary.

[0141] Example 1

[0142] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] Autonomous mobile devices are required to accurately recognize no-entry areas and move around safely and efficiently. However, with conventional technology, the determination of no-entry areas and updating of map data are often done manually, making it difficult to respond in real time. In addition, managing the retention period for information about no-entry areas and taking appropriate evasive action are difficult, which can lead to reduced operational efficiency and safety for autonomous mobile devices.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0145] In this invention, the server includes means for transmitting image data acquired by an imaging device mounted on the autonomous mobile device, means for generating the image data as a prompt, means for transmitting the prompt to a generative artificial intelligence model and receiving feedback regarding no-entry areas, means for analyzing the feedback and reflecting the no-entry areas on a map, means for managing information regarding the no-entry areas based on a retention period, and means for recalculating the route of the autonomous mobile device to avoid the no-entry areas. This enables the autonomous mobile device to autonomously recognize no-entry areas, update map data in real time, and take efficient avoidance actions.

[0146] An "autonomous mobile device" is a device equipped with sensors, cameras, and various control programs that moves and operates on its own judgment.

[0147] An "imaging device" is hardware for capturing images or video, such as a camera or depth sensor.

[0148] "Image data" is data that digitally represents visual information acquired by an imaging device.

[0149] A "prompt" is a data structure containing instructions or questions that are input to a generative AI model.

[0150] A "generative artificial intelligence model" is an AI system that uses machine learning and deep learning technologies to analyze and make predictions based on input data.

[0151] "Feedback" refers to the analysis results and response data returned as a response from the generative AI model.

[0152] A "no-entry area" is a specific spatial region to which access is restricted for the purpose of managing the safety of people and equipment and improving work efficiency.

[0153] "Reflecting on the map" means updating geographical data based on the acquired information and accurately showing the new information.

[0154] "Retention period" refers to the period for which information is considered valid, and means maintaining and managing the information until that period has elapsed.

[0155] "Recalculating a route" refers to recalculating the optimal route between the current location and the destination.

[0156] "3D point cloud data" is a collection of a series of points that make up a space, and each point is expressed in a 3D coordinate system (x, y, z).

[0157] The present invention relates to a system for an autonomous mobile device to autonomously determine a no-entry area and take action to avoid the area. Implementing the present invention requires the following specific steps.

[0158] Server Operation

[0159] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. Specific hardware used is a camera module (image capture device) and a depth sensor (a device for acquiring 3D point cloud data).

[0160] First, the server periodically receives camera images sent from the autonomous mobile device. This image data is temporarily stored in a buffer. The reception process is often performed using an HTTP POST request.

[0161] Next, the server generates a prompt to analyze the received image data. This prompt contains the path to the image file and the text "Is this area off-limits?". For example, the following prompt is generated:

[0162] Image: / tmp / captured_image.jpg

[0163] Q: Is this area off-limits?

[0164] The generated prompt is sent to the API endpoint of the generative AI model. After being sent to the API endpoint, the server waits for feedback from the generative AI model. The feedback includes information on whether the area is a no-entry area.

[0165] The server receives feedback from the AI ​​model and analyzes its contents. For example, it may receive feedback such as "This area is off-limits." If this feedback is analyzed and it is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. Specifically, it links the 3D point cloud data acquired from the depth camera with 2D images to perform detailed mapping.

[0166] Finally, the server manages the information about restricted areas based on a retention period. For example, if the generative AI model instructs the server to keep the information for one day, the server will register the information in the database and schedule a deletion task one day later.

[0167] Terminal (autonomous mobile device) operation

[0168] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. Specifically, it performs the following operations.

[0169] First, the autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in the device's memory.

[0170] Next, the autonomous mobile device transmits the stored image data to the server. If the transmission is unsuccessful, a retry mechanism is used to attempt retransmission.

[0171] Receives feedback from the server and obtains information on restricted areas. Analyzes the information, recalculates the route based on the restricted area information, and creates an alternative route to the destination. If no suitable route is found, notifies the administrator.

[0172] User (facility administrator) actions

[0173] Users use the management dashboard to monitor and manage the entire system. The specific actions of users are as follows:

[0174] First, the user accesses the management dashboard and monitors the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0175] Next, the user enters settings for prohibiting entry to specific areas on the dashboard and sends them to the server. Information on the specified areas and period is sent to the server and reflected throughout the system.

[0176] Finally, the user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route or operation schedule.

[0177] Specific examples

[0178] For example, the following scenario is possible:

[0179] The server receives the image data from the "Autonomous Operation Sensor" and saves it as / tmp / captured_image.jpg. It generates a prompt containing the received image and formats it as follows:

[0180] Image: / tmp / captured_image.jpg

[0181] Q: Is this area off-limits?

[0182] This prompt is sent to the API endpoint of the generative AI model, and feedback is received stating, "This area is prohibited from entry." The location of the no-entry sign is converted into 3D coordinates, and the map data of the autonomous mobile device is updated. The generative AI model also registers the retention period information, "This information is valid for one day," in the database, and schedules a deletion task one day later.

[0183] The autonomous mobile device captures images using a camera module within the facility and sends them to the server via an HTTP POST request. It then receives feedback from the server, calculates a new route based on the information on "no entry areas," and continues moving toward its destination.

[0184] Users log in to the dashboard, monitor the operation status of the autonomous mobile device in real time, set up temporary prohibitions on certain areas, and send the information to the server. The system checks whether the robot is avoiding the prohibited areas and adjusts its route and schedule as necessary.

[0185] The above describes a specific embodiment of the invention. This system enables an autonomous mobile device to autonomously recognize no-entry areas, update map data in real time, and take efficient avoidance actions.

[0186] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0187] Server Operation

[0188] Step 1:

[0189] The server receives camera images from the autonomous mobile device.

[0190] Specific behavior:

[0191] The server receives the HTTP POST request and acquires the camera image data. The received image is temporarily stored in buffer memory.

[0192] Input: Camera image data sent from an autonomous mobile device.

[0193] Output: The image data will be saved in / tmp / captured_image.jpg.

[0194] Step 2:

[0195] The server generates a prompt to analyze the received image data.

[0196] Specific behavior:

[0197] The server generates a JSON prompt containing the file path of the received image and the text "Is this area off-limits?"

[0198] Input: The camera image saved as / tmp / captured_image.jpg and the question text.

[0199] Output: The generated prompt (e.g., {"image": " / tmp / captured_image.jpg", "question": "Is this area off-limits?"}).

[0200] Step 3:

[0201] The server sends the generated prompts to the generating artificial intelligence model.

[0202] Specific behavior:

[0203] The server sends an HTTP POST request to the API endpoint of the generated AI model.

[0204] Input: The generated prompt.

[0205] Output: The request sent to the API endpoint of the generative AI model.

[0206] Step 4:

[0207] The server receives and analyzes feedback from the generative AI model.

[0208] Specific behavior:

[0209] The server receives the HTTP response from the generated AI model and analyzes the feedback content, which is "No entry."

[0210] Input: Feedback from the generative AI model.

[0211] Output: No entry area determination result.

[0212] Step 5:

[0213] If the server determines that the area is a no-entry area, it calculates the spatial coordinates of the area and updates the map data.

[0214] Specific behavior:

[0215] The 3D point cloud data acquired by the depth camera is linked to the 2D images to calculate the precise location of no-entry areas and update the map data.

[0216] Input: No-entry area determination results, 3D point cloud data, and 2D image data.

[0217] Output: Updated map data.

[0218] Step 6:

[0219] The server manages the information about the no-entry areas based on a retention period.

[0220] Specific behavior:

[0221] Receive retention period information (e.g., "valid for 1 day") from the generated AI model and register it in the database. Schedule a deletion task after the specified period.

[0222] Input: No entry area information and retention period information.

[0223] Output: No entry area information registered in the database with retention period.

[0224] Terminal (autonomous mobile device) operation

[0225] Step 1:

[0226] The autonomous mobile device captures camera images at regular intervals.

[0227] Specific behavior:

[0228] The autonomous mobile device periodically activates the camera module to capture image data and temporarily store it in memory.

[0229] Input: A periodic timer event.

[0230] Output: Captured camera image data.

[0231] Step 2:

[0232] The autonomous mobile device transmits the captured images to a server.

[0233] Specific behavior:

[0234] The autonomous mobile device sends the stored image data to the server using an HTTP POST request. If the transmission is unsuccessful, the device attempts to retransmit the data using a retry mechanism.

[0235] Input: Captured camera image data.

[0236] Output: Image data sent to the server.

[0237] Step 3:

[0238] The autonomous mobile device receives feedback from the server.

[0239] Specific behavior:

[0240] Wait for feedback sent from the server and analyze it as soon as it is received.

[0241] Input: Feedback data from the server.

[0242] Output: Parsed feedback data.

[0243] Step 4:

[0244] The autonomous mobile device takes evasive action based on information about no-entry areas.

[0245] Specific behavior:

[0246] The system recalculates routes based on information about restricted areas and plans alternative routes to the destination. If no suitable route is found, it notifies the administrator.

[0247] Input: Parsed no-go area information.

[0248] Output: Recalculated route information or notification to administrator.

[0249] User (facility administrator) actions

[0250] Step 1:

[0251] Users access an administrative dashboard to monitor the status of their systems.

[0252] Specific behavior:

[0253] Through the dashboard, users can check the current location of the autonomous mobile device, its operating status, and the status of no-entry areas in real time.

[0254] Input: Dashboard access information and system status data.

[0255] Output: Visualized information showing the current state.

[0256] Step 2:

[0257] The user inputs settings to prohibit entry into specific areas.

[0258] Specific behavior:

[0259] The user designates a specific area as off-limits on the dashboard and sends that information to the server.

[0260] Input: User configuration input.

[0261] Output: No entry configuration information sent to the server.

[0262] Step 3:

[0263] The user checks the operating status of the autonomous mobile device.

[0264] Specific behavior:

[0265] The user monitors whether the autonomous mobile device recognizes no-entry areas and takes appropriate avoidance actions, adjusting the route or schedule as necessary.

[0266] Input: Dashboard traffic data.

[0267] Output: Check results and any necessary adjustments.

[0268] As described above, by providing detailed explanations for each processing step, including specific operations and inputs and outputs, we aim to ensure that the operation of this system is accurate and efficient.

[0269] (Application example 1)

[0270] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0271] Conventional autonomous mobile devices have limited capabilities for detecting and avoiding restricted areas, which has led to problems in ensuring safety, particularly in complex industrial facilities. There is also a risk that workers may mistakenly enter restricted or dangerous areas, which could lead to workplace accidents. Therefore, there is a need to improve safety by improving the ability of autonomous mobile devices to recognize restricted areas and by adding a real-time warning function to workers.

[0272] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0273] In this invention, the server includes means for periodically receiving camera images and temporarily storing them in a buffer, means for generating prompts for analyzing the received image data, means for sending the generated prompts to a generative artificial intelligence model and receiving feedback regarding no-entry areas, means for analyzing the feedback and mapping the no-entry areas, means for managing information regarding no-entry areas based on a retention period, and means for notifying smart devices worn by workers in industrial facilities in real time of information regarding no-entry areas and dangerous areas. This not only enables autonomous mobile devices to more accurately recognize and avoid no-entry areas, but also allows workers to be immediately warned of no-entry areas and dangerous areas, significantly improving safety.

[0274] An "autonomous mobile device" is a mechanical device that uses sensors to recognize its surrounding environment, automatically plans a route to its destination, and moves accordingly.

[0275] "Camera image" refers to visual information acquired by a camera mounted on an autonomous mobile device.

[0276] A "prompt" is a question generated based on camera images, and is text information provided to a generative artificial intelligence model to determine no-entry areas.

[0277] A "generative artificial intelligence model" is a system with machine learning algorithms used to generate answers to prompts.

[0278] "Feedback" is a response output from the generative artificial intelligence model, and includes information about no-entry areas.

[0279] "Mapping" is the process of converting the location of the no-entry area into spatial coordinates and recording them as map data for the autonomous mobile device.

[0280] "Retention period" refers to the period for which information regarding restricted areas is stored within the system.

[0281] "Industrial facility" refers to buildings and premises where manufacturing or other industrial activity is carried out.

[0282] "Smart equipment" refers to wearable devices worn by factory workers that have the ability to receive and display information about restricted and dangerous areas in real time.

[0283] "Real-time notification" means instantly transmitting information about restricted or dangerous areas to users.

[0284] A "hazardous area" is an area that poses a potential hazard to personnel or equipment.

[0285] The present invention relates to a system that enables autonomous mobile devices to autonomously determine restricted areas and take evasive action. In particular, the system can improve worker safety by linking with smart devices worn by workers in industrial facilities and notifying them of restricted or dangerous areas in real time.

[0286] Server Operation

[0287] The server receives camera images sent from the autonomous mobile device, analyzes them, and determines which areas are off-limits. The server uses the following hardware and software:

[0288] Hardware: high performance servers, storage devices, network interfaces

[0289] Software: image analysis algorithms, generative AI models, databases

[0290] The server first periodically receives camera images sent from the autonomous mobile device and stores them in a buffer. Next, it generates a prompt to analyze the received image data. This prompt includes the image data and the question, "Is this area off-limits?" The generated prompt is sent to the generative AI model, which receives feedback regarding the off-limits area. The received feedback is analyzed, and if it is determined to be an off-limits area, the location is converted into spatial coordinates and mapped. Finally, it sets a retention period for information regarding off-limits areas and manages them in a database.

[0291] Terminal (autonomous mobile device) operation

[0292] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. The hardware and software of the terminal are as follows.

[0293] Hardware: Camera module, communication module, control unit

[0294] Software: Image capture software, communication protocols, path planning algorithms

[0295] The autonomous mobile device first captures images at regular intervals using a camera module. The captured image data is stored in memory and sent to a server. After receiving feedback from the server and obtaining information about no-entry areas, the device calculates an alternative route to the destination and continues moving while avoiding the no-entry areas.

[0296] User (factory workers and managers) actions

[0297] Factory workers and managers manage and monitor the system. The hardware and software used are as follows:

[0298] Hardware: Smart helmet, dashboard display

[0299] Software: Real-time monitoring application, notification system, configuration management interface

[0300] Factory workers wear smart helmets and receive real-time notifications of restricted or dangerous areas. For example, a message such as "This area is prohibited. Please avoid it" appears on the helmet's display. Managers can also use a dashboard to monitor the operation of the autonomous mobile devices and adjust zone settings and schedules.

[0301] Specific examples

[0302] Specific examples are shown below.

[0303] The server periodically receives and analyzes "image data within the facility."

[0304] An example of a prompt statement is: "Is this area off-limits?"

[0305] Workers can take immediate action when the message "Danger area. Please evacuate" appears on the display inside the smart helmet.

[0306] This provides a system that not only enables autonomous mobile devices to accurately recognize and avoid no-entry areas, but also improves the safety of factory workers.

[0307] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0308] Step 1:

[0309] The server periodically receives image data from the autonomous mobile device. The input is camera image data, which is temporarily stored in the server's buffer. This buffering allows for smooth subsequent data analysis.

[0310] Step 2:

[0311] The server generates a prompt based on the received image data. The input is the image data saved in the previous step, and the output is the prompt text. The server generates a prompt in JSON format that includes the image data and the text "Is this area off-limits?"

[0312] Step 3:

[0313] The server sends the generated prompt to the generative AI model and receives feedback. The input is the generated prompt, and the output is feedback from the generative AI model. The server determines whether the area is off-limits based on this feedback.

[0314] Step 4:

[0315] The server analyzes the feedback and maps the no-entry areas to spatial coordinates. The input is the feedback data from the generative AI model, and the output is the spatial coordinate data of the no-entry areas. The server links the 2D image data with the 3D point cloud data to obtain detailed location information.

[0316] Step 5:

[0317] The server manages information about restricted areas based on retention periods. The input is the spatial coordinate data of the restricted areas and retention period information, and the output is the updated database state. The server sets a schedule for deleting or updating the data after the set period.

[0318] Step 6:

[0319] The autonomous mobile device captures camera images at regular intervals. The input is real-time video data, and the output is captured still images. The images are stored in memory via the camera module.

[0320] Step 7:

[0321] The autonomous mobile device sends the captured image to the server. The input is the captured image data, and the output is an HTTP request to the server. The image data is sent to the server using the communication module.

[0322] Step 8:

[0323] The autonomous mobile device receives feedback from the server and acquires the no-entry area information. The input is the feedback data sent from the server, and the output is the analyzed no-entry area information. The information is acquired through the communication module.

[0324] Step 9:

[0325] The autonomous mobile device calculates a new route based on the no-entry area information and continues moving toward the destination. The input is the no-entry area information and the output is a new navigation route. A path planning algorithm is used to calculate the avoidance route.

[0326] Step 10:

[0327] Users wear smart devices and receive real-time notifications of restricted or dangerous areas. The input is a warning message from the server, and the output is the warning information displayed on the smart device's display. Users can immediately check this information and take evasive action.

[0328] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0329] The present invention relates to a system that enables an autonomous mobile device to autonomously determine no-entry areas and take evasive action. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it achieves more flexible and user-friendly operation management. Below, specific embodiments of each element and its operation for implementing the present invention will be described.

[0330] Server Operation

[0331] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. It also acquires user emotional information and adjusts the settings of the entire system. The specific operation of the server is described below.

[0332] 1. Receiving camera images

[0333] The server receives camera images periodically transmitted from the autonomous mobile device, and the received image data is temporarily stored in a buffer.

[0334] 2. Prompt Generation

[0335] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[0336] 3. Send to generative AI model

[0337] The server sends the generated prompt to the generative AI model and waits for feedback, which includes information about the no-go area.

[0338] 4. Feedback Analysis

[0339] The feedback from the generative artificial intelligence model is analyzed to determine whether the area is off-limits.

[0340] 5. Mapping No-Go Areas

[0341] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. A depth camera is used to link 3D point cloud data with 2D images to perform detailed mapping.

[0342] 6. Information Retention Period Management

[0343] Information about restricted areas is retained for a set period based on feedback from the generative AI model, and is then deleted or updated from the database after the period expires.

[0344] 7. Emotion Recognition with Emotion Engine

[0345] To obtain the user's emotional information, the emotion engine analyzes the user's speech and facial expressions, and adjusts the no-entry zone settings accordingly.

[0346] Terminal (autonomous mobile device) operation

[0347] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action. It also adjusts its operation status based on feedback from the emotion engine. The specific operation of the autonomous mobile device is described below.

[0348] 1. Camera image capture

[0349] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[0350] 2. Sending images

[0351] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[0352] 3. Receiving Instructions

[0353] Receive feedback from the server and obtain information about restricted areas.

[0354] 4. Taking evasive action

[0355] Based on the information about the no-entry areas, the autonomous mobile device recalculates an alternative route to the destination and avoids the no-entry areas.

[0356] 5. Receiving emotional feedback

[0357] It receives feedback from an emotion engine and adjusts its speed and route based on the user's stress level and emotional state.

[0358] User (facility administrator) actions

[0359] The user manages and monitors the system. The specific actions of the user are described below.

[0360] 1. System Monitoring

[0361] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0362] 2. Change settings

[0363] Users can enter settings to prohibit entry to specific areas from the dashboard and send them to the server. Information on the designated area and period is sent to the server and reflected throughout the system.

[0364] 3. Check operation

[0365] The user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route and operation schedule.

[0366] 4. Monitoring and setting emotion data

[0367] Users can monitor data from the emotion engine and adjust system settings based on their emotional state, for example slowing down the robot's speed if stress levels are high.

[0368] Specific examples

[0369] The present invention will be described below with reference to specific examples.

[0370] Server example

[0371] 1. Receiving camera images

[0372] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[0373] 2. Generate and send prompts

[0374] Generate a prompt containing the received image and format it in JSON format.

[0375] json

[0376] {

[0377] "image": " / tmp / captured_image.jpg",

[0378] "question": "Is this area off-limits?"

[0379] }

[0380] This prompt is sent to the API endpoint of the generative AI model, and feedback is received saying, "This area is off-limits."

[0381] 3. Mapping No-Go Areas

[0382] The location of the no entry sign is converted into 3D coordinates and the map data of the autonomous mobile device is updated.

[0383] 4. Information Retention Period Management

[0384] The retention period information from the generative AI model, "This information is valid for one day," is registered in the database, and a deletion task is scheduled for one day later.

[0385] 5. Emotion Recognition by Emotion Engine

[0386] The system analyzes the user's speech and facial expressions and adjusts the settings of no-entry areas based on that information.

[0387] Examples of autonomous mobile devices

[0388] 1. Camera image capture

[0389] The autonomous mobile device uses a camera module to capture images within the facility.

[0390] 2. Sending images

[0391] Send the image data to the server using an HTTP POST request.

[0392] 3. Receiving and parsing instructions

[0393] Receive feedback from the server that the area is off-limits and analyze the content.

[0394] 4. Taking evasive action

[0395] Based on information about restricted areas, a new route is calculated and travel to the destination is continued.

[0396] 5. Receiving emotional feedback

[0397] It receives feedback from the emotion engine and adjusts its speed and route according to stress levels.

[0398] User Examples

[0399] 1. System Monitoring

[0400] Log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[0401] 2. Change settings

[0402] Settings are made to temporarily prohibit entry to specific areas and sent to the server.

[0403] 3. Check operation

[0404] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[0405] 4. Monitoring and setting emotion data

[0406] The system settings are adjusted based on emotional information obtained from the emotion engine, optimizing the robot's operation according to the user's stress level.

[0407] The processing flow will be explained below.

[0408] Server Processing Steps

[0409] Step 1:

[0410] The server waits for and receives camera images sent by the autonomous mobile device via HTTP requests, and temporarily stores the received image data in a buffer.

[0411] Step 2:

[0412] The server passes the received camera image to the prompt generation module, which generates a prompt containing the image and the text "Is this area off-limits?". The prompt is then formatted into JSON format to be sent to the generative AI model.

[0413] Step 3:

[0414] The server sends the formatted prompt via an HTTPS request to the API endpoint of the generative AI model and waits for feedback from the generative AI model.

[0415] Step 4:

[0416] The server receives feedback from the generative AI model in JSON format, including information such as "no entry areas," and analyzes the content to determine whether the area is a no entry area.

[0417] Step 5:

[0418] The server references spatial data linked to the camera's viewpoint to obtain coordinate information for the restricted area based on the feedback. It uses a depth camera or stereo method to link 3D data with 2D images and calculates detailed spatial coordinates.

[0419] Step 6:

[0420] The server adds the calculated spatial coordinates to the robot's map data and displays it on the map as a new no-entry area.

[0421] Step 7:

[0422] The server stores the retention period information provided in the feedback of the generative AI model in a database, manages the retention period of the no-entry area information, and schedules a task to update the information when the retention period expires.

[0423] Step 8:

[0424] The server uses an emotion engine to analyze the user's speech and facial expressions, and adjusts the settings for no-entry areas and other operational parameters based on the analysis results.

[0425] Processing steps of terminal (autonomous mobile device)

[0426] Step 1:

[0427] The autonomous mobile device captures images from the camera module at regular intervals and temporarily stores them in memory.

[0428] Step 2:

[0429] The autonomous mobile device sends the saved image to a specified server endpoint via an HTTP POST request. It also has a retry mechanism until the transmission is successful, ensuring communication stability.

[0430] Step 3:

[0431] The autonomous mobile device receives feedback from the server, analyzes the content, and obtains information about restricted areas.

[0432] Step 4:

[0433] The autonomous mobile device uses the received coordinate information of the no-entry area to autonomously calculate a new route and avoid the no-entry area.

[0434] Step 5:

[0435] The autonomous mobile device instructs the navigation module to find an avoidance route and continues moving to the destination. If the device cannot find a route to the destination, it notifies the administrator.

[0436] Step 6:

[0437] The autonomous mobile device receives feedback from the emotion engine and adjusts its travel speed and route based on the user's stress level and emotional state.

[0438] User (facility manager) processing steps

[0439] Step 1:

[0440] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0441] Step 2:

[0442] Users can set specific areas to be off-limits as needed by entering the area's coordinates and the off-limits period into the form on the management screen and pressing the "Set" button to send the information to the server.

[0443] Step 3:

[0444] The user checks whether the autonomous mobile device is properly avoiding restricted areas and adjusts the robot's route and operation schedule as necessary.

[0445] Step 4:

[0446] Based on the emotional information obtained from the emotion engine, the user can adjust the system settings according to the user's emotional state, for example, slowing down the robot's speed if the user's stress level is high.

[0447] As a concrete example, a specific scenario for each step will be explained below.

[0448] Server example

[0449] Step 1:

[0450] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[0451] Step 2:

[0452] The prompt generation module generates a prompt in JSON format that includes the received image and the question "Is this area off-limits?"

[0453] Step 3:

[0454] The generated prompt is sent via an HTTPS request to the API endpoint of the generated artificial intelligence model.

[0455] Step 4:

[0456] The generative AI model receives feedback such as "This area is off-limits" and parses it in JSON format.

[0457] Step 5:

[0458] Based on the feedback, the 3D coordinates of the no-entry area are calculated from the image and updated in the map data of the autonomous mobile device.

[0459] Step 6:

[0460] The calculated coordinates of the no-entry area are added to the robot's map data, and the mapping is updated.

[0461] Step 7:

[0462] Store retention period information in a database and schedule a task to update the information once the retention period has expired.

[0463] Step 8:

[0464] The emotion engine is used to analyze the user's speech and facial expressions, and the system adjusts no-entry zone settings and operating parameters based on the results.

[0465] Examples of autonomous mobile devices

[0466] Step 1:

[0467] The autonomous mobile device uses a camera module to capture images within the facility.

[0468] Step 2:

[0469] The captured image data is sent to the server via an HTTP POST request.

[0470] Step 3:

[0471] It receives feedback from the server that the area is off-limits and analyzes the content.

[0472] Step 4:

[0473] Based on information about restricted areas, the system calculates a new route and autonomously avoids restricted areas.

[0474] Step 5:

[0475] The system instructs the navigation module to find an avoidance route and continues moving to the destination. If the system is unable to find a route to the destination, it notifies the administrator.

[0476] Step 6:

[0477] It receives feedback from the emotion engine and adjusts its speed and route according to the user's stress level.

[0478] User Examples

[0479] Step 1:

[0480] Users log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[0481] Step 2:

[0482] Set up a specific area to be temporarily off-limits, enter the coordinates and period, and send it to the server.

[0483] Step 3:

[0484] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[0485] Step 4:

[0486] The system settings are adjusted based on emotional information obtained from the emotion engine, optimizing the robot's operation according to the user's stress level.

[0487] Example 2

[0488] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0489] Conventional autonomous mobile devices have difficulty in accurately identifying no-entry areas and taking evasive action. Furthermore, they lacked operational control that took into account the user's emotional state, resulting in a lack of flexibility and user-friendliness. Furthermore, they lacked management of the retention period for no-entry area information and detailed mapping of spatial coordinates.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0491] In this invention, the server includes a means for receiving camera images, a means for generating prompts, a means for transmitting the prompts to a generative AI model and receiving feedback, a means for analyzing the feedback and mapping no-entry areas, a means for managing no-entry area information based on a retention period, and a means for acquiring emotion information using an emotion engine and adjusting system settings based on the emotion information. This enables the autonomous mobile device to appropriately determine and avoid no-entry areas, realizing flexible and user-friendly operation management based on the user's emotional state.

[0492] An "autonomous mobile device" is a device that moves autonomously without external instructions and performs designated tasks using specific programs and sensors.

[0493] A "camera" is a device that converts light into an electrical signal and records it as image data.

[0494] "Image data" is a digital representation of visual information captured by a camera.

[0495] A "prompt" is text data that includes specific image data along with questions or instructions for the generative AI model.

[0496] A "generative AI model" is a computer model that uses artificial intelligence techniques to generate feedback based on given input data (prompts).

[0497] "Feedback" is a response from a generative AI model that contains useful information or instructions for the system.

[0498] A "no-entry area" is a specific area where autonomous mobile devices are prohibited from entering.

[0499] "Mapping" is the process of locating a specific area or element on a coordinate system and representing that information visually or digitally.

[0500] "Retention Period" means the period for which collected information is stored and valid in our database.

[0501] An "emotion engine" is a technology that analyzes a user's speech and facial expressions to determine their emotional state.

[0502] "System settings" are various configurations and adjustment items that determine the operation and behavior of the entire system.

[0503] This invention is a system in which an autonomous mobile device autonomously determines no-entry areas and takes evasive action. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, flexible and user-friendly operation management is realized. Below, specific embodiments of each element and its operation for implementing this invention are described.

[0504] Server Operation

[0505] The server receives camera images from the autonomous mobile device, analyzes them, maps out no-entry areas, and acquires user emotional information to adjust the overall system settings.

[0506] Hardware and Software

[0507] The server uses a computer server equipped with a high-performance processor and large amount of memory, and the HTTP protocol is used for communication. SSDs are used for disk storage, enabling high-speed reading and writing of image data. A cloud-based service equipped with a generative AI model is used for analysis.

[0508] Data processing and calculation

[0509] The server receives camera images periodically sent from the autonomous mobile device. The received image data is temporarily stored in a buffer and a prompt is generated based on the image data. For example, a prompt such as "Is this area off-limits?" is added.

[0510] Examples of prompts:

[0511] Image: / tmp / captured_image.jpg

[0512] Q: Is this area off-limits?

[0513] This prompt is sent to the generative AI model, which receives feedback regarding the no-entry area. The feedback contains information about the no-entry area, which is then analyzed to determine whether it is a no-entry area. If it is determined to be a no-entry area, the spatial coordinates of that area are calculated and the map data of the autonomous mobile device is updated.

[0514] Furthermore, information about restricted areas is managed based on a retention period, and is deleted or updated from the database after the specified period has expired. An emotion engine is used to analyze the user's speech and facial expressions to understand the user's emotional state. Based on this, system settings such as restricted area settings and operating speeds are adjusted.

[0515] Operation of autonomous mobile device

[0516] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action, and also adjusts its operation status based on feedback from the emotion engine.

[0517] Hardware and Software

[0518] Autonomous mobile devices use mobile devices equipped with camera modules, processors, communication modules, etc. For communication, they use high-speed communication technologies such as Wi-Fi and 5G.

[0519] Data processing and calculation

[0520] The autonomous mobile device captures camera images at regular intervals and temporarily stores the image data in its memory. The stored images are then sent to the server using an HTTP POST request. After receiving feedback about prohibited areas from the server, the device recalculates an alternative route to the destination based on that information, avoiding the prohibited areas.

[0521] In addition, based on feedback from the emotion engine, the system adjusts the travel speed and route according to the user's stress level and emotional state. For example, if the user is feeling stressed, the system will slow down the travel speed.

[0522] User (facility administrator) actions

[0523] Users manage and monitor the system.

[0524] Hardware and Software

[0525] Users use devices (e.g., PCs, tablets, smartphones) that can access the management dashboard. The dashboard uses an internet browser and is designed with an emphasis on real-time performance.

[0526] Data processing and calculation

[0527] Users can monitor the current location and operational status of their autonomous mobile devices, as well as the status of restricted area settings, in real time from the management dashboard. If necessary, they can configure specific areas to be restricted from entry from the dashboard and send that information to the server, where the settings are reflected throughout the system.

[0528] The user also confirms that the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the route and operation schedule are adjusted. Furthermore, data from the emotion engine is monitored, and system settings are adjusted according to the user's emotional state. For example, if the user's stress level is high, the robot's operation speed may be slowed down.

[0529] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0530] Server Processing Steps

[0531] Step 1: Receiving camera images

[0532] The server receives image data periodically sent from the autonomous mobile device via an HTTP POST request. This image data is temporarily saved in a buffer file called " / tmp / captured_image.jpg."

[0533] Input: Image data sent from the autonomous mobile device (e.g., / tmp / captured_image.jpg)

[0534] Output: Temporarily saved image data

[0535] Step 2: Generate the prompt

[0536] The server generates a prompt using the received image data, for example adding the text "Is this area off-limits?" along with the path of the received image.

[0537] Input: Received image data (e.g., / tmp / captured_image.jpg)

[0538] Output: Generated prompt text (e.g. "Image: / tmp / captured_image.jpg Question: Is this area off-limits?")

[0539] Step 3: Send to generative AI model

[0540] The server sends the generated prompt to the API endpoint of the generative AI model via an HTTP POST request and waits for feedback from the generative AI model.

[0541] Input: Generated prompt text

[0542] Output: Feedback from the generative AI model

[0543] Step 4: Analyze the feedback

[0544] The server analyzes the feedback received from the generative AI model. For example, if it receives feedback such as "This area is off-limits," it analyzes the content and determines whether it is a off-limits area.

[0545] Input: Feedback from the generative AI model

[0546] Output: No entry area determination result

[0547] Step 5: Map out the no-go areas

[0548] If the server determines that an area is a no-entry area, it calculates the spatial coordinates of that area. It uses a depth camera to link 3D point cloud data with 2D image data to perform detailed mapping. It then updates the map data of the autonomous mobile device.

[0549] Input: No-entry area determination results, 3D point cloud data, 2D image data

[0550] Output: Updated map data

[0551] Step 6: Managing information retention period

[0552] The server manages information about restricted areas based on a retention period. It registers the information in the database based on the retention period information from the generative AI model, and deletes or updates it after the specified period has expired.

[0553] Input: No entry area information, retention period information

[0554] Output: Database update status

[0555] Step 7: Emotion Recognition with the Emotion Engine

[0556] The server uses an emotion engine to analyze the user's speech and facial expressions to understand their emotional state, and adjusts system settings accordingly. For example, if the user is dissatisfied, the server revises the settings for no-entry areas.

[0557] Input: User speech and facial expression data

[0558] Output: System settings adjustment status

[0559] Processing steps of an autonomous mobile device

[0560] Step 1: Capture a camera image

[0561] The autonomous mobile device captures camera images at regular intervals, and the image data is temporarily stored in memory.

[0562] Input: Visual information in the environment

[0563] Output: Captured image data

[0564] Step 2: Sending images

[0565] The autonomous mobile device sends the stored images to the server. It has a retry mechanism until the transmission is successful, ensuring communication stability. It uses HTTP POST requests.

[0566] Input: Captured image data

[0567] Output: Image data sent to the server

[0568] Step 3: Receiving instructions

[0569] The autonomous mobile device receives feedback from the server and acquires information about no-entry areas. If the feedback is "This area is no-entry," the device performs processing based on that information.

[0570] Input: Feedback from the server

[0571] Output: No entry area information

[0572] Step 4: Take evasive action

[0573] The autonomous mobile device calculates a new route based on the information about the restricted area and continues moving toward the destination. It selects a different route to avoid the restricted area.

[0574] Input: No entry area information

[0575] Output: Updated route information as a result of taking evasive action

[0576] Step 5: Receiving Emotional Feedback

[0577] The autonomous mobile device receives feedback from the emotion engine and adjusts its travel speed and route according to the user's stress level and emotional state, for example, slowing down if the user is feeling stressed.

[0578] Input: Feedback from the emotion engine

[0579] Output: Adjusted speed and route information

[0580] User processing steps

[0581] Step 1: Monitor your system

[0582] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0583] Input: Various operation information, system setting information

[0584] Output: Displayed operation status and setting status

[0585] Step 2: Change the settings

[0586] Users can enter settings to restrict access to specific areas through the dashboard and send them to the server, where the information is processed and reflected throughout the system.

[0587] Input: Information about the settings you want to change

[0588] Output: Settings reflected on the server

[0589] Step 3: Check operation

[0590] The user confirms that the autonomous mobile device properly recognizes the restricted area and takes evasive action, adjusting the route and operation schedule as necessary.

[0591] Input: Operation status information of autonomous mobile device

[0592] Output: Confirmed operational status, adjusted routes and schedules

[0593] Step 4: Monitoring and Configuring Emotion Data

[0594] Users monitor data from the emotion engine and adjust system settings based on their emotional state. For example, if a user's stress level is high, the robot's speed may be slowed down.

[0595] Input: Emotion engine data

[0596] Output: Adjusted system settings

[0597] (Application example 2)

[0598] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0599] Conventional autonomous mobile devices have had difficulty in properly recognizing and avoiding no-entry areas. Furthermore, they lacked the means to dynamically manage no-entry area information and adjust their operating speed and route based on the emotional state of the user (delivery worker or customer). This has made it difficult to operate efficiently and safely in delivery operations, resulting in a decline in user satisfaction.

[0600] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for managing information about no-entry areas based on a retention period, means for supporting delivery robot operation management on a smart device application, means for calculating an alternative route to avoid the no-entry area and instructing the delivery robot to take that route, and means for analyzing the user's emotional state and adjusting the operation speed and route based on that information. This makes it possible to dynamically manage no-entry areas and realize flexible operation adjustments according to the user's emotional state.

[0601] An "autonomous mobile device" is a device that moves autonomously and performs designated tasks.

[0602] An "imaging device" is a device that includes a camera and a sensor for acquiring image data.

[0603] A "prompt" is information containing questions or instructions about image data to be input to a generative AI model.

[0604] A "generative artificial intelligence model" is an AI model that generates no-entry area determinations and other feedback based on input prompts.

[0605] "Feedback" refers to the judgment results or responses obtained from a generative AI model.

[0606] "Mapping" is the act of placing no-entry areas on a map as spatial information based on the acquired data.

[0607] The "retention period" is the period during which information regarding no-entry areas is valid, and once this period has elapsed, the information will be updated or deleted.

[0608] A "delivery robot" is a robot that automatically makes deliveries to people, vehicles, etc.

[0609] "Smart devices" are devices with advanced computing power and communication functions, and primarily refer to smartphones and tablets.

[0610] An "application" is a program or software that runs on a smart device.

[0611] An "alternate route" is a newly calculated route to avoid a no-entry area.

[0612] "Emotional state" refers to the user's emotional or psychological state, primarily stress level and satisfaction.

[0613] "Operation speed" refers to the speed at which the delivery robot moves.

[0614] A "route" is the path a delivery robot travels.

[0615] "Analysis" is the process of interpreting and making sense of data and information.

[0616] This invention relates to a system that enables autonomous mobile devices to autonomously determine no-entry areas and take evasive action. In addition, by combining this with an emotion engine that recognizes the user's emotions, it realizes more flexible and user-friendly operation management.

[0617] Server Operation

[0618] The server recognizes and maps no-entry areas using the following hardware and software:

[0619] 1. Receiving camera images

[0620] The server receives camera images periodically transmitted from the autonomous mobile device, and temporarily stores the received image data in a buffer.

[0621] 2. Prompt Generation

[0622] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[0623] Example prompt sentence:

[0624] Image: / path / to / image.jpg

[0625] Q: Is this area off-limits?

[0626] 3. Send to generative AI model

[0627] The server sends the generated prompt to the generative artificial intelligence model and waits for feedback, which includes information about the no-go area.

[0628] 4. Feedback Analysis

[0629] The feedback obtained from the generative AI model is analyzed to determine no-entry areas. The generative AI model is used for image analysis.

[0630] 5. Mapping No-Go Areas

[0631] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device, closely linking 2D image data with 3D point cloud data.

[0632] 6. Information Retention Period Management

[0633] No-entry area information is retained for a set period based on feedback from the generative AI model, after which it is deleted or updated from the database.

[0634] 7. Emotion Recognition with Emotion Engine

[0635] The emotion engine analyzes the user's speech and facial expressions and adjusts operation management based on that information.

[0636] Terminal (autonomous mobile device) operation

[0637] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action, and also adjusts its operation status based on feedback from the emotion engine.

[0638] 1. Camera image capture

[0639] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[0640] 2. Sending images

[0641] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[0642] 3. Receiving Instructions

[0643] Receive feedback from the server and obtain information about restricted areas.

[0644] 4. Taking evasive action

[0645] Based on the information about the no-entry areas, the autonomous mobile device recalculates an alternative route to the destination and avoids the no-entry areas.

[0646] 5. Receiving emotional feedback

[0647] It receives feedback from an emotion engine and adjusts its speed and route based on the user's stress level and emotional state.

[0648] User (delivery person or customer) behavior

[0649] The user manages and monitors the system.

[0650] 1. System Monitoring

[0651] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zones, in real time.

[0652] 2. Change settings

[0653] Users enter settings to prohibit entry to specific areas from the dashboard and send them to the server.

[0654] 3. Check operation

[0655] Users can check whether their autonomous mobile device is properly recognizing no-entry areas and taking evasive action, and adjust their route or operation schedule as necessary.

[0656] 4. Monitoring and setting emotion data

[0657] Users can monitor data from the emotion engine and adjust system settings based on their emotional state, for example slowing down the delivery robot if their stress levels are high.

[0658] As described above, the present invention provides means for making the operation management of autonomous mobile devices more efficient and safe.

[0659] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0660] Step 1:

[0661] The autonomous mobile device (terminal) captures camera images at regular intervals. The captured image data is temporarily stored in memory. The input is image data from the camera sensor, and the output is an image file in memory.

[0662] Step 2:

[0663] The device sends the image stored in memory to the server using an HTTP POST request. The input here is the image file in memory, and the output is the image data sent to the server. To ensure communication stability, a retry mechanism is included until the transmission is successful.

[0664] Step 3:

[0665] The server receives camera images sent from the autonomous mobile device and temporarily stores them in a buffer. The input is the image data from the device, and the output is the image data in the buffer.

[0666] Step 4:

[0667] The server generates a prompt to analyze the received image data. The prompt contains the image along with the text "Is this area off-limits?". The input is the image data in the buffer and a predefined question, and the output is the generated prompt.

[0668] Step 5:

[0669] The server sends the generated prompt to the AI ​​model and waits for feedback. The input is the prompt sentence, and the output is feedback about the no-entry area.

[0670] Step 6:

[0671] The server analyzes the feedback obtained from the generative AI model and determines the no-entry areas. The input is the feedback data returned from the generative AI model, and the output is the no-entry area determination result.

[0672] Step 7:

[0673] If a no-entry area is determined, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. The inputs are the no-entry area determination result, 2D image data, and 3D point cloud data, and the output is the updated map data.

[0674] Step 8:

[0675] The server manages information about restricted areas based on a retention period. The input is the information about restricted areas and the retention period setting, and the output is the data to be updated or deleted after the retention period has passed.

[0676] Step 9:

[0677] The server uses an emotion engine to analyze the user's speech and facial expressions, and adjusts operation management based on that information. The input is the user's speech data and facial expression data, and the output is analyzed emotional information. Based on this emotional information, the operation speed and route are adjusted.

[0678] Step 10:

[0679] The device receives feedback from the server and recalculates an alternative route based on the information on no-entry areas. The input is the no-entry area information from the server, and the output is the newly calculated alternative route.

[0680] Step 11:

[0681] The device continues moving to the destination according to the recalculated alternative route. The input is the new route information, and the output is the result of the movement.

[0682] Step 12:

[0683] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device in real time. The input is real-time data stored on the server, and the output is the operation status displayed on the user's screen.

[0684] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0685] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0686] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0687] [Second embodiment]

[0688] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0689] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0690] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0691] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0692] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0693] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0694] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0695] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0696] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0697] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0698] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0699] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0700] The present invention relates to a system for an autonomous mobile device to autonomously determine a no-entry area and take action to avoid it. The elements required to implement the present invention and their operations will be described below in detail.

[0701] Server Operation

[0702] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. The specific operation of the server is described below.

[0703] 1. Receiving camera images

[0704] The server receives camera images periodically transmitted from the autonomous mobile device, and the received image data is temporarily stored in a buffer.

[0705] 2. Prompt Generation

[0706] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[0707] 3. Send to generative AI model

[0708] The server sends the generated prompt to the artificial intelligence model and waits for a response, which includes information about the no-entry area.

[0709] 4. Feedback Analysis

[0710] The feedback from the generative artificial intelligence model is analyzed to determine whether the area is a no-entry area.

[0711] 5. Mapping No-Go Areas

[0712] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. A depth camera is used to link 3D point cloud data with 2D images to perform detailed mapping.

[0713] 6. Information Retention Period Management

[0714] Information about restricted areas is retained for a set period based on feedback from the generative AI model, and is then deleted or updated from the database after the period expires.

[0715] Terminal (autonomous mobile device) operation

[0716] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. The specific operation of the autonomous mobile device is described below.

[0717] 1. Camera image capture

[0718] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[0719] 2. Sending images

[0720] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[0721] 3. Receiving Instructions

[0722] Receive feedback from the server and obtain information about restricted areas.

[0723] 4. Taking evasive action

[0724] Based on the information on the restricted area, the autonomous mobile device recalculates an alternative route to the destination and avoids the restricted area. If it is unable to plot a route to the destination, it notifies the administrator.

[0725] User (facility administrator) actions

[0726] The user manages and monitors the system. The specific actions of the user are described below.

[0727] 1. System Monitoring

[0728] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0729] 2. Change settings

[0730] Users can enter settings to prohibit entry to specific areas from the dashboard and send them to the server. Information on the designated area and period is sent to the server and reflected throughout the system.

[0731] 3. Check operation

[0732] The user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route and operation schedule.

[0733] Specific examples

[0734] The present invention will be described below with reference to specific examples.

[0735] Server example

[0736] 1. Receiving camera images

[0737] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[0738] 2. Prompt Generation

[0739] Generate a prompt containing the received image and format it in JSON format.

[0740] json

[0741] {

[0742] "image": " / tmp / captured_image.jpg",

[0743] "question": "Is this area off-limits?"

[0744] }

[0745] 3. Sending to the generative AI model and receiving feedback

[0746] This prompt is sent to the API endpoint of the generative AI model, and feedback is received saying, "This area is off-limits."

[0747] 4. Mapping No-Go Areas

[0748] The location of the no entry sign is converted into 3D coordinates and the map data of the autonomous mobile device is updated.

[0749] 5. Information Retention Period Management

[0750] The retention period information from the generative AI model, "This information is valid for one day," is registered in the database, and a deletion task is scheduled for one day later.

[0751] Examples of autonomous mobile devices

[0752] 1. Camera image capture

[0753] The autonomous mobile device uses a camera module to capture images within the facility.

[0754] 2. Sending images

[0755] Send the image data to the server using an HTTP POST request.

[0756] 3. Receiving Instructions

[0757] Receive feedback from the server that the area is off-limits and analyze the content.

[0758] 4. Taking evasive action

[0759] Based on information about restricted areas, a new route is calculated and travel to the destination is continued.

[0760] User Examples

[0761] 1. System Monitoring

[0762] Log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[0763] 2. Change settings

[0764] Settings are made to temporarily prohibit entry to specific areas and sent to the server.

[0765] 3. Check operation

[0766] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[0767] The processing flow will be explained below.

[0768] Server Processing Steps

[0769] Step 1:

[0770] The server waits for and receives camera images sent by the autonomous mobile device via HTTP requests, and temporarily stores the received image data in a buffer.

[0771] Step 2:

[0772] The server passes the received camera image to the prompt generation module, which generates a prompt containing the image and the text "Is this area off-limits?". The prompt is then formatted into JSON format to be sent to the generative AI model.

[0773] Step 3:

[0774] The server sends the formatted prompt via an HTTPS request to the API endpoint of the generative AI model and waits for feedback from the generative AI model.

[0775] Step 4:

[0776] The server receives feedback from the generative AI model in JSON format, including information such as "no entry areas," and analyzes the content to determine whether the area is a no entry area.

[0777] Step 5:

[0778] The server references spatial data linked to the camera's viewpoint to obtain coordinate information for the restricted area based on the feedback. It uses a depth camera or stereo method to link 3D data with 2D images and calculates detailed spatial coordinates.

[0779] Step 6:

[0780] The server adds the calculated spatial coordinates to the robot's map data and maps it as a new no-entry area.

[0781] Step 7:

[0782] The server stores the retention period information provided in the feedback of the generative AI model in a database, manages the retention period of the no-entry area information, and schedules a task to update the information when the retention period expires.

[0783] Processing steps of terminal (autonomous mobile device)

[0784] Step 1:

[0785] The autonomous mobile device captures images from the camera module at regular intervals and temporarily stores them in memory.

[0786] Step 2:

[0787] The autonomous mobile device sends the saved image to a specified server endpoint via an HTTP POST request. It also has a retry mechanism until the transmission is successful, ensuring communication stability.

[0788] Step 3:

[0789] The autonomous mobile device receives feedback from the server, analyzes the content, and obtains information about restricted areas.

[0790] Step 4:

[0791] The autonomous mobile device uses the received coordinate information of the no-entry area to autonomously calculate a new route and avoid the no-entry area.

[0792] Step 5:

[0793] The autonomous mobile device instructs the navigation module to find an avoidance route and continues moving to the destination. If the device cannot find a route to the destination, it notifies the administrator.

[0794] User (facility manager) processing steps

[0795] Step 1:

[0796] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0797] Step 2:

[0798] Users can set specific areas to be off-limits as needed by entering the area's coordinates and the off-limits period into the form on the management screen and pressing the "Set" button to send the information to the server.

[0799] Step 3:

[0800] The user checks whether the autonomous mobile device is properly avoiding restricted areas and adjusts the robot's route and operation schedule as necessary.

[0801] Example 1

[0802] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0803] Autonomous mobile devices are required to accurately recognize no-entry areas and move around safely and efficiently. However, with conventional technology, the determination of no-entry areas and updating of map data are often done manually, making it difficult to respond in real time. In addition, managing the retention period for information about no-entry areas and taking appropriate evasive action are difficult, which can lead to reduced operational efficiency and safety for autonomous mobile devices.

[0804] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0805] In this invention, the server includes means for transmitting image data acquired by an imaging device mounted on the autonomous mobile device, means for generating the image data as a prompt, means for transmitting the prompt to a generative artificial intelligence model and receiving feedback regarding no-entry areas, means for analyzing the feedback and reflecting the no-entry areas on a map, means for managing information regarding the no-entry areas based on a retention period, and means for recalculating the route of the autonomous mobile device to avoid the no-entry areas. This enables the autonomous mobile device to autonomously recognize no-entry areas, update map data in real time, and take efficient avoidance actions.

[0806] An "autonomous mobile device" is a device equipped with sensors, cameras, and various control programs that moves and operates on its own judgment.

[0807] An "imaging device" is hardware for capturing images or video, such as a camera or depth sensor.

[0808] "Image data" is data that digitally represents visual information acquired by an imaging device.

[0809] A "prompt" is a data structure containing instructions or questions that are input to a generative AI model.

[0810] A "generative artificial intelligence model" is an AI system that uses machine learning and deep learning technologies to analyze and make predictions based on input data.

[0811] "Feedback" refers to the analysis results and response data returned as a response from the generative AI model.

[0812] A "no-entry area" is a specific spatial region to which access is restricted for the purpose of managing the safety of people and equipment and improving work efficiency.

[0813] "Reflecting on the map" means updating geographical data based on the acquired information and accurately showing the new information.

[0814] "Retention period" refers to the period for which information is considered valid, and means maintaining and managing the information until that period has elapsed.

[0815] "Recalculating a route" refers to recalculating the optimal route between the current location and the destination.

[0816] "3D point cloud data" is a collection of a series of points that make up a space, and each point is expressed in a 3D coordinate system (x, y, z).

[0817] The present invention relates to a system for an autonomous mobile device to autonomously determine a no-entry area and take action to avoid the area. Implementing the present invention requires the following specific steps.

[0818] Server Operation

[0819] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. Specific hardware used is a camera module (image capture device) and a depth sensor (a device for acquiring 3D point cloud data).

[0820] First, the server periodically receives camera images sent from the autonomous mobile device. This image data is temporarily stored in a buffer. The reception process is often performed using an HTTP POST request.

[0821] Next, the server generates a prompt to analyze the received image data. This prompt contains the path to the image file and the text "Is this area off-limits?". For example, the following prompt is generated:

[0822] Image: / tmp / captured_image.jpg

[0823] Q: Is this area off-limits?

[0824] The generated prompt is sent to the API endpoint of the generative AI model. After being sent to the API endpoint, the server waits for feedback from the generative AI model. The feedback includes information on whether the area is a no-entry area.

[0825] The server receives feedback from the AI ​​model and analyzes its contents. For example, it may receive feedback such as "This area is off-limits." If this feedback is analyzed and it is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. Specifically, it links the 3D point cloud data acquired from the depth camera with 2D images to perform detailed mapping.

[0826] Finally, the server manages the information about restricted areas based on a retention period. For example, if the generative AI model instructs the server to keep the information for one day, the server will register the information in the database and schedule a deletion task one day later.

[0827] Terminal (autonomous mobile device) operation

[0828] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. Specifically, it performs the following operations.

[0829] First, the autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in the device's memory.

[0830] Next, the autonomous mobile device transmits the stored image data to the server. If the transmission is unsuccessful, a retry mechanism is used to attempt retransmission.

[0831] Receives feedback from the server and obtains information on restricted areas. Analyzes the information, recalculates the route based on the restricted area information, and creates an alternative route to the destination. If no suitable route is found, notifies the administrator.

[0832] User (facility administrator) actions

[0833] Users use the management dashboard to monitor and manage the entire system. The specific actions of users are as follows:

[0834] First, the user accesses the management dashboard and monitors the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[0835] Next, the user enters settings for prohibiting entry to specific areas on the dashboard and sends them to the server. Information on the specified areas and period is sent to the server and reflected throughout the system.

[0836] Finally, the user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route or operation schedule.

[0837] Specific examples

[0838] For example, the following scenario is possible:

[0839] The server receives the image data from the "Autonomous Operation Sensor" and saves it as / tmp / captured_image.jpg. It generates a prompt containing the received image and formats it as follows:

[0840] Image: / tmp / captured_image.jpg

[0841] Q: Is this area off-limits?

[0842] This prompt is sent to the API endpoint of the generative AI model, and feedback is received stating, "This area is prohibited from entry." The location of the no-entry sign is converted into 3D coordinates, and the map data of the autonomous mobile device is updated. The generative AI model also registers the retention period information, "This information is valid for one day," in the database, and schedules a deletion task one day later.

[0843] The autonomous mobile device captures images using a camera module within the facility and sends them to the server via an HTTP POST request. It then receives feedback from the server, calculates a new route based on the information on "no entry areas," and continues moving toward its destination.

[0844] Users log in to the dashboard, monitor the operation status of the autonomous mobile device in real time, set up temporary prohibitions on certain areas, and send the information to the server. The system checks whether the robot is avoiding the prohibited areas and adjusts its route and schedule as necessary.

[0845] The above describes a specific embodiment of the invention. This system enables an autonomous mobile device to autonomously recognize no-entry areas, update map data in real time, and take efficient avoidance actions.

[0846] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0847] Server Operation

[0848] Step 1:

[0849] The server receives camera images from the autonomous mobile device.

[0850] Specific behavior:

[0851] The server receives the HTTP POST request and acquires the camera image data. The received image is temporarily stored in buffer memory.

[0852] Input: Camera image data sent from an autonomous mobile device.

[0853] Output: The image data will be saved in / tmp / captured_image.jpg.

[0854] Step 2:

[0855] The server generates a prompt to analyze the received image data.

[0856] Specific behavior:

[0857] The server generates a JSON prompt containing the file path of the received image and the text "Is this area off-limits?"

[0858] Input: The camera image saved as / tmp / captured_image.jpg and the question text.

[0859] Output: The generated prompt (e.g., {"image": " / tmp / captured_image.jpg", "question": "Is this area off-limits?"}).

[0860] Step 3:

[0861] The server sends the generated prompts to the generating artificial intelligence model.

[0862] Specific behavior:

[0863] The server sends an HTTP POST request to the API endpoint of the generated AI model.

[0864] Input: The generated prompt.

[0865] Output: The request sent to the API endpoint of the generative AI model.

[0866] Step 4:

[0867] The server receives and analyzes feedback from the generative AI model.

[0868] Specific behavior:

[0869] The server receives the HTTP response from the generated AI model and analyzes the feedback content, which is "No entry."

[0870] Input: Feedback from the generative AI model.

[0871] Output: No entry area determination result.

[0872] Step 5:

[0873] If the server determines that the area is a no-entry area, it calculates the spatial coordinates of the area and updates the map data.

[0874] Specific behavior:

[0875] The 3D point cloud data acquired by the depth camera is linked to the 2D images to calculate the precise location of no-entry areas and update the map data.

[0876] Input: No-entry area determination results, 3D point cloud data, and 2D image data.

[0877] Output: Updated map data.

[0878] Step 6:

[0879] The server manages the information about the no-entry areas based on a retention period.

[0880] Specific behavior:

[0881] Receive retention period information (e.g., "valid for 1 day") from the generated AI model and register it in the database. Schedule a deletion task after the specified period.

[0882] Input: No entry area information and retention period information.

[0883] Output: No entry area information registered in the database with retention period.

[0884] Terminal (autonomous mobile device) operation

[0885] Step 1:

[0886] The autonomous mobile device captures camera images at regular intervals.

[0887] Specific behavior:

[0888] The autonomous mobile device periodically activates the camera module to capture image data and temporarily store it in memory.

[0889] Input: A periodic timer event.

[0890] Output: Captured camera image data.

[0891] Step 2:

[0892] The autonomous mobile device transmits the captured images to a server.

[0893] Specific behavior:

[0894] The autonomous mobile device sends the stored image data to the server using an HTTP POST request. If the transmission is unsuccessful, the device attempts to retransmit the data using a retry mechanism.

[0895] Input: Captured camera image data.

[0896] Output: Image data sent to the server.

[0897] Step 3:

[0898] The autonomous mobile device receives feedback from the server.

[0899] Specific behavior:

[0900] Wait for feedback sent from the server and analyze it as soon as it is received.

[0901] Input: Feedback data from the server.

[0902] Output: Parsed feedback data.

[0903] Step 4:

[0904] The autonomous mobile device takes evasive action based on information about no-entry areas.

[0905] Specific behavior:

[0906] The system recalculates routes based on information about restricted areas and plans alternative routes to the destination. If no suitable route is found, it notifies the administrator.

[0907] Input: Parsed no-go area information.

[0908] Output: Recalculated route information or notification to administrator.

[0909] User (facility administrator) actions

[0910] Step 1:

[0911] Users access an administrative dashboard to monitor the status of their systems.

[0912] Specific behavior:

[0913] Through the dashboard, users can check the current location of the autonomous mobile device, its operating status, and the status of no-entry areas in real time.

[0914] Input: Dashboard access information and system status data.

[0915] Output: Visualized information showing the current state.

[0916] Step 2:

[0917] The user inputs settings to prohibit entry into specific areas.

[0918] Specific behavior:

[0919] The user designates a specific area as off-limits on the dashboard and sends that information to the server.

[0920] Input: User configuration input.

[0921] Output: No entry configuration information sent to the server.

[0922] Step 3:

[0923] The user checks the operating status of the autonomous mobile device.

[0924] Specific behavior:

[0925] The user monitors whether the autonomous mobile device recognizes no-entry areas and takes appropriate avoidance actions, adjusting the route or schedule as necessary.

[0926] Input: Dashboard traffic data.

[0927] Output: Check results and any necessary adjustments.

[0928] As described above, by providing detailed explanations for each processing step, including specific operations and inputs and outputs, we aim to ensure that the operation of this system is accurate and efficient.

[0929] (Application example 1)

[0930] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0931] Conventional autonomous mobile devices have limited capabilities for detecting and avoiding restricted areas, which has led to problems in ensuring safety, particularly in complex industrial facilities. There is also a risk that workers may mistakenly enter restricted or dangerous areas, which could lead to workplace accidents. Therefore, there is a need to improve safety by improving the ability of autonomous mobile devices to recognize restricted areas and by adding a real-time warning function to workers.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0933] In this invention, the server includes means for periodically receiving camera images and temporarily storing them in a buffer, means for generating prompts for analyzing the received image data, means for sending the generated prompts to a generative artificial intelligence model and receiving feedback regarding no-entry areas, means for analyzing the feedback and mapping the no-entry areas, means for managing information regarding no-entry areas based on a retention period, and means for notifying smart devices worn by workers in industrial facilities in real time of information regarding no-entry areas and dangerous areas. This not only enables autonomous mobile devices to more accurately recognize and avoid no-entry areas, but also allows workers to be immediately warned of no-entry areas and dangerous areas, significantly improving safety.

[0934] An "autonomous mobile device" is a mechanical device that uses sensors to recognize its surrounding environment, automatically plans a route to its destination, and moves accordingly.

[0935] "Camera image" refers to visual information acquired by a camera mounted on an autonomous mobile device.

[0936] A "prompt" is a question generated based on camera images, and is text information provided to a generative artificial intelligence model to determine no-entry areas.

[0937] A "generative artificial intelligence model" is a system with machine learning algorithms used to generate answers to prompts.

[0938] "Feedback" is a response output from the generative artificial intelligence model, and includes information about no-entry areas.

[0939] "Mapping" is the process of converting the location of the no-entry area into spatial coordinates and recording them as map data for the autonomous mobile device.

[0940] "Retention period" refers to the period for which information regarding restricted areas is stored within the system.

[0941] "Industrial facility" refers to buildings and premises where manufacturing or other industrial activity is carried out.

[0942] "Smart equipment" refers to wearable devices worn by factory workers that have the ability to receive and display information about restricted and dangerous areas in real time.

[0943] "Real-time notification" means instantly transmitting information about restricted or dangerous areas to users.

[0944] A "hazardous area" is an area that poses a potential hazard to personnel or equipment.

[0945] The present invention relates to a system that enables autonomous mobile devices to autonomously determine restricted areas and take evasive action. In particular, the system can improve worker safety by linking with smart devices worn by workers in industrial facilities and notifying them of restricted or dangerous areas in real time.

[0946] Server Operation

[0947] The server receives camera images sent from the autonomous mobile device, analyzes them, and determines which areas are off-limits. The server uses the following hardware and software:

[0948] Hardware: high performance servers, storage devices, network interfaces

[0949] Software: image analysis algorithms, generative AI models, databases

[0950] The server first periodically receives camera images sent from the autonomous mobile device and stores them in a buffer. Next, it generates a prompt to analyze the received image data. This prompt includes the image data and the question, "Is this area off-limits?" The generated prompt is sent to the generative AI model, which receives feedback regarding the off-limits area. The received feedback is analyzed, and if it is determined to be an off-limits area, the location is converted into spatial coordinates and mapped. Finally, it sets a retention period for information regarding off-limits areas and manages them in a database.

[0951] Terminal (autonomous mobile device) operation

[0952] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. The hardware and software of the terminal are as follows.

[0953] Hardware: Camera module, communication module, control unit

[0954] Software: Image capture software, communication protocols, path planning algorithms

[0955] The autonomous mobile device first captures images at regular intervals using a camera module. The captured image data is stored in memory and sent to a server. After receiving feedback from the server and obtaining information about no-entry areas, the device calculates an alternative route to the destination and continues moving while avoiding the no-entry areas.

[0956] User (factory workers and managers) actions

[0957] Factory workers and managers manage and monitor the system. The hardware and software used are as follows:

[0958] Hardware: Smart helmet, dashboard display

[0959] Software: Real-time monitoring application, notification system, configuration management interface

[0960] Factory workers wear smart helmets and receive real-time notifications of restricted or dangerous areas. For example, a message such as "This area is prohibited. Please avoid it" appears on the helmet's display. Managers can also use a dashboard to monitor the operation of the autonomous mobile devices and adjust zone settings and schedules.

[0961] Specific examples

[0962] Specific examples are shown below.

[0963] The server periodically receives and analyzes "image data within the facility."

[0964] An example of a prompt statement is: "Is this area off-limits?"

[0965] Workers can take immediate action when the message "Danger area. Please evacuate" appears on the display inside the smart helmet.

[0966] This provides a system that not only enables autonomous mobile devices to accurately recognize and avoid no-entry areas, but also improves the safety of factory workers.

[0967] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0968] Step 1:

[0969] The server periodically receives image data from the autonomous mobile device. The input is camera image data, which is temporarily stored in the server's buffer. This buffering allows for smooth subsequent data analysis.

[0970] Step 2:

[0971] The server generates a prompt based on the received image data. The input is the image data saved in the previous step, and the output is the prompt text. The server generates a prompt in JSON format that includes the image data and the text "Is this area off-limits?"

[0972] Step 3:

[0973] The server sends the generated prompt to the generative AI model and receives feedback. The input is the generated prompt, and the output is feedback from the generative AI model. The server determines whether the area is off-limits based on this feedback.

[0974] Step 4:

[0975] The server analyzes the feedback and maps the no-entry areas to spatial coordinates. The input is the feedback data from the generative AI model, and the output is the spatial coordinate data of the no-entry areas. The server links the 2D image data with the 3D point cloud data to obtain detailed location information.

[0976] Step 5:

[0977] The server manages information about restricted areas based on retention periods. The input is the spatial coordinate data of the restricted areas and retention period information, and the output is the updated database state. The server sets a schedule for deleting or updating the data after the set period.

[0978] Step 6:

[0979] The autonomous mobile device captures camera images at regular intervals. The input is real-time video data, and the output is captured still images. The images are stored in memory via the camera module.

[0980] Step 7:

[0981] The autonomous mobile device sends the captured image to the server. The input is the captured image data, and the output is an HTTP request to the server. The image data is sent to the server using the communication module.

[0982] Step 8:

[0983] The autonomous mobile device receives feedback from the server and acquires the no-entry area information. The input is the feedback data sent from the server, and the output is the analyzed no-entry area information. The information is acquired through the communication module.

[0984] Step 9:

[0985] The autonomous mobile device calculates a new route based on the no-entry area information and continues moving toward the destination. The input is the no-entry area information and the output is a new navigation route. A path planning algorithm is used to calculate the avoidance route.

[0986] Step 10:

[0987] Users wear smart devices and receive real-time notifications of restricted or dangerous areas. The input is a warning message from the server, and the output is the warning information displayed on the smart device's display. Users can immediately check this information and take evasive action.

[0988] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0989] The present invention relates to a system that enables an autonomous mobile device to autonomously determine no-entry areas and take evasive action. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it achieves more flexible and user-friendly operation management. Below, specific embodiments of each element and its operation for implementing the present invention will be described.

[0990] Server Operation

[0991] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. It also acquires user emotional information and adjusts the settings of the entire system. The specific operation of the server is described below.

[0992] 1. Receiving camera images

[0993] The server receives camera images periodically transmitted from the autonomous mobile device, and the received image data is temporarily stored in a buffer.

[0994] 2. Prompt Generation

[0995] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[0996] 3. Send to generative AI model

[0997] The server sends the generated prompt to the generative AI model and waits for feedback, which includes information about the no-go area.

[0998] 4. Feedback Analysis

[0999] The feedback from the generative artificial intelligence model is analyzed to determine whether the area is off-limits.

[1000] 5. Mapping No-Go Areas

[1001] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. A depth camera is used to link 3D point cloud data with 2D images to perform detailed mapping.

[1002] 6. Information Retention Period Management

[1003] Information about restricted areas is retained for a set period based on feedback from the generative AI model, and is then deleted or updated from the database after the period expires.

[1004] 7. Emotion Recognition with Emotion Engine

[1005] To obtain the user's emotional information, the emotion engine analyzes the user's speech and facial expressions, and adjusts the no-entry zone settings accordingly.

[1006] Terminal (autonomous mobile device) operation

[1007] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action. It also adjusts its operation status based on feedback from the emotion engine. The specific operation of the autonomous mobile device is described below.

[1008] 1. Camera image capture

[1009] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[1010] 2. Sending images

[1011] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[1012] 3. Receiving Instructions

[1013] Receive feedback from the server and obtain information about restricted areas.

[1014] 4. Taking evasive action

[1015] Based on the information about the no-entry areas, the autonomous mobile device recalculates an alternative route to the destination and avoids the no-entry areas.

[1016] 5. Receiving emotional feedback

[1017] It receives feedback from an emotion engine and adjusts its speed and route based on the user's stress level and emotional state.

[1018] User (facility administrator) actions

[1019] The user manages and monitors the system. The specific actions of the user are described below.

[1020] 1. System Monitoring

[1021] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1022] 2. Change settings

[1023] Users can enter settings to prohibit entry to specific areas from the dashboard and send them to the server. Information on the designated area and period is sent to the server and reflected throughout the system.

[1024] 3. Check operation

[1025] The user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route and operation schedule.

[1026] 4. Monitoring and setting emotion data

[1027] Users can monitor data from the emotion engine and adjust system settings based on their emotional state, for example slowing down the robot's speed if stress levels are high.

[1028] Specific examples

[1029] The present invention will be described below with reference to specific examples.

[1030] Server example

[1031] 1. Receiving camera images

[1032] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[1033] 2. Generate and send prompts

[1034] Generate a prompt containing the received image and format it in JSON format.

[1035] json

[1036] {

[1037] "image": " / tmp / captured_image.jpg",

[1038] "question": "Is this area off-limits?"

[1039] }

[1040] This prompt is sent to the API endpoint of the generative AI model, and feedback is received saying, "This area is off-limits."

[1041] 3. Mapping No-Go Areas

[1042] The location of the no entry sign is converted into 3D coordinates and the map data of the autonomous mobile device is updated.

[1043] 4. Information Retention Period Management

[1044] The retention period information from the generative AI model, "This information is valid for one day," is registered in the database, and a deletion task is scheduled for one day later.

[1045] 5. Emotion Recognition by Emotion Engine

[1046] The system analyzes the user's speech and facial expressions and adjusts the settings of no-entry areas based on that information.

[1047] Examples of autonomous mobile devices

[1048] 1. Camera image capture

[1049] The autonomous mobile device uses a camera module to capture images within the facility.

[1050] 2. Sending images

[1051] Send the image data to the server using an HTTP POST request.

[1052] 3. Receiving and parsing instructions

[1053] Receive feedback from the server that the area is off-limits and analyze the content.

[1054] 4. Taking evasive action

[1055] Based on information about restricted areas, a new route is calculated and travel to the destination is continued.

[1056] 5. Receiving emotional feedback

[1057] It receives feedback from the emotion engine and adjusts its speed and route according to stress levels.

[1058] User Examples

[1059] 1. System Monitoring

[1060] Log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[1061] 2. Change settings

[1062] Settings are made to temporarily prohibit entry to specific areas and sent to the server.

[1063] 3. Check operation

[1064] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[1065] 4. Monitoring and setting emotion data

[1066] The system settings are adjusted based on emotional information obtained from the emotion engine, optimizing the robot's operation according to the user's stress level.

[1067] The processing flow will be explained below.

[1068] Server Processing Steps

[1069] Step 1:

[1070] The server waits for and receives camera images sent by the autonomous mobile device via HTTP requests, and temporarily stores the received image data in a buffer.

[1071] Step 2:

[1072] The server passes the received camera image to the prompt generation module, which generates a prompt containing the image and the text "Is this area off-limits?". The prompt is then formatted into JSON format to be sent to the generative AI model.

[1073] Step 3:

[1074] The server sends the formatted prompt via an HTTPS request to the API endpoint of the generative AI model and waits for feedback from the generative AI model.

[1075] Step 4:

[1076] The server receives feedback from the generative AI model in JSON format, including information such as "no entry areas," and analyzes the content to determine whether the area is a no entry area.

[1077] Step 5:

[1078] The server references spatial data linked to the camera's viewpoint to obtain coordinate information for the restricted area based on the feedback. It uses a depth camera or stereo method to link 3D data with 2D images and calculates detailed spatial coordinates.

[1079] Step 6:

[1080] The server adds the calculated spatial coordinates to the robot's map data and displays it on the map as a new no-entry area.

[1081] Step 7:

[1082] The server stores the retention period information provided in the feedback of the generative AI model in a database, manages the retention period of the no-entry area information, and schedules a task to update the information when the retention period expires.

[1083] Step 8:

[1084] The server uses an emotion engine to analyze the user's speech and facial expressions, and adjusts the settings for no-entry areas and other operational parameters based on the analysis results.

[1085] Processing steps of terminal (autonomous mobile device)

[1086] Step 1:

[1087] The autonomous mobile device captures images from the camera module at regular intervals and temporarily stores them in memory.

[1088] Step 2:

[1089] The autonomous mobile device sends the saved image to a specified server endpoint via an HTTP POST request. It also has a retry mechanism until the transmission is successful, ensuring communication stability.

[1090] Step 3:

[1091] The autonomous mobile device receives feedback from the server, analyzes the content, and obtains information about restricted areas.

[1092] Step 4:

[1093] The autonomous mobile device uses the received coordinate information of the no-entry area to autonomously calculate a new route and avoid the no-entry area.

[1094] Step 5:

[1095] The autonomous mobile device instructs the navigation module to find an avoidance route and continues moving to the destination. If the device cannot find a route to the destination, it notifies the administrator.

[1096] Step 6:

[1097] The autonomous mobile device receives feedback from the emotion engine and adjusts its travel speed and route based on the user's stress level and emotional state.

[1098] User (facility manager) processing steps

[1099] Step 1:

[1100] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1101] Step 2:

[1102] Users can set specific areas to be off-limits as needed by entering the area's coordinates and the off-limits period into the form on the management screen and pressing the "Set" button to send the information to the server.

[1103] Step 3:

[1104] The user checks whether the autonomous mobile device is properly avoiding restricted areas and adjusts the robot's route and operation schedule as necessary.

[1105] Step 4:

[1106] Based on the emotional information obtained from the emotion engine, the user can adjust the system settings according to the user's emotional state, for example, slowing down the robot's speed if the user's stress level is high.

[1107] As a concrete example, a specific scenario for each step will be explained below.

[1108] Server example

[1109] Step 1:

[1110] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[1111] Step 2:

[1112] The prompt generation module generates a prompt in JSON format that includes the received image and the question "Is this area off-limits?"

[1113] Step 3:

[1114] The generated prompt is sent via an HTTPS request to the API endpoint of the generated artificial intelligence model.

[1115] Step 4:

[1116] The generative AI model receives feedback such as "This area is off-limits" and parses it in JSON format.

[1117] Step 5:

[1118] Based on the feedback, the 3D coordinates of the no-entry area are calculated from the image and updated in the map data of the autonomous mobile device.

[1119] Step 6:

[1120] The calculated coordinates of the no-entry area are added to the robot's map data, and the mapping is updated.

[1121] Step 7:

[1122] Store retention period information in a database and schedule a task to update the information once the retention period has expired.

[1123] Step 8:

[1124] The emotion engine is used to analyze the user's speech and facial expressions, and the system adjusts no-entry zone settings and operating parameters based on the results.

[1125] Examples of autonomous mobile devices

[1126] Step 1:

[1127] The autonomous mobile device uses a camera module to capture images within the facility.

[1128] Step 2:

[1129] The captured image data is sent to the server via an HTTP POST request.

[1130] Step 3:

[1131] It receives feedback from the server that the area is off-limits and analyzes the content.

[1132] Step 4:

[1133] Based on information about restricted areas, the system calculates a new route and autonomously avoids restricted areas.

[1134] Step 5:

[1135] The system instructs the navigation module to find an avoidance route and continues moving to the destination. If the system is unable to find a route to the destination, it notifies the administrator.

[1136] Step 6:

[1137] It receives feedback from the emotion engine and adjusts its speed and route according to the user's stress level.

[1138] User Examples

[1139] Step 1:

[1140] Users log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[1141] Step 2:

[1142] Set up a specific area to be temporarily off-limits, enter the coordinates and period, and send it to the server.

[1143] Step 3:

[1144] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[1145] Step 4:

[1146] The system settings are adjusted based on emotional information obtained from the emotion engine, optimizing the robot's operation according to the user's stress level.

[1147] Example 2

[1148] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1149] Conventional autonomous mobile devices have difficulty in accurately identifying no-entry areas and taking evasive action. Furthermore, they lacked operational control that took into account the user's emotional state, resulting in a lack of flexibility and user-friendliness. Furthermore, they lacked management of the retention period for no-entry area information and detailed mapping of spatial coordinates.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1151] In this invention, the server includes a means for receiving camera images, a means for generating prompts, a means for transmitting the prompts to a generative AI model and receiving feedback, a means for analyzing the feedback and mapping no-entry areas, a means for managing no-entry area information based on a retention period, and a means for acquiring emotion information using an emotion engine and adjusting system settings based on the emotion information. This enables the autonomous mobile device to appropriately determine and avoid no-entry areas, realizing flexible and user-friendly operation management based on the user's emotional state.

[1152] An "autonomous mobile device" is a device that moves autonomously without external instructions and performs designated tasks using specific programs and sensors.

[1153] A "camera" is a device that converts light into an electrical signal and records it as image data.

[1154] "Image data" is a digital representation of visual information captured by a camera.

[1155] A "prompt" is text data that includes specific image data along with questions or instructions for the generative AI model.

[1156] A "generative AI model" is a computer model that uses artificial intelligence techniques to generate feedback based on given input data (prompts).

[1157] "Feedback" is a response from a generative AI model that contains useful information or instructions for the system.

[1158] A "no-entry area" is a specific area where autonomous mobile devices are prohibited from entering.

[1159] "Mapping" is the process of locating a specific area or element on a coordinate system and representing that information visually or digitally.

[1160] "Retention Period" means the period for which collected information is stored and valid in our database.

[1161] An "emotion engine" is a technology that analyzes a user's speech and facial expressions to determine their emotional state.

[1162] "System settings" are various configurations and adjustment items that determine the operation and behavior of the entire system.

[1163] This invention is a system in which an autonomous mobile device autonomously determines no-entry areas and takes evasive action. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, flexible and user-friendly operation management is realized. Below, specific embodiments of each element and its operation for implementing this invention are described.

[1164] Server Operation

[1165] The server receives camera images from the autonomous mobile device, analyzes them, maps out no-entry areas, and acquires user emotional information to adjust the overall system settings.

[1166] Hardware and Software

[1167] The server uses a computer server equipped with a high-performance processor and large amount of memory, and the HTTP protocol is used for communication. SSDs are used for disk storage, enabling high-speed reading and writing of image data. A cloud-based service equipped with a generative AI model is used for analysis.

[1168] Data processing and calculation

[1169] The server receives camera images periodically sent from the autonomous mobile device. The received image data is temporarily stored in a buffer and a prompt is generated based on the image data. For example, a prompt such as "Is this area off-limits?" is added.

[1170] Examples of prompts:

[1171] Image: / tmp / captured_image.jpg

[1172] Q: Is this area off-limits?

[1173] This prompt is sent to the generative AI model, which receives feedback regarding the no-entry area. The feedback contains information about the no-entry area, which is then analyzed to determine whether it is a no-entry area. If it is determined to be a no-entry area, the spatial coordinates of that area are calculated and the map data of the autonomous mobile device is updated.

[1174] Furthermore, information about restricted areas is managed based on a retention period, and is deleted or updated from the database after the specified period has expired. An emotion engine is used to analyze the user's speech and facial expressions to understand the user's emotional state. Based on this, system settings such as restricted area settings and operating speeds are adjusted.

[1175] Operation of autonomous mobile device

[1176] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action, and also adjusts its operation status based on feedback from the emotion engine.

[1177] Hardware and Software

[1178] Autonomous mobile devices use mobile devices equipped with camera modules, processors, communication modules, etc. For communication, they use high-speed communication technologies such as Wi-Fi and 5G.

[1179] Data processing and calculation

[1180] The autonomous mobile device captures camera images at regular intervals and temporarily stores the image data in its memory. The stored images are then sent to the server using an HTTP POST request. After receiving feedback about prohibited areas from the server, the device recalculates an alternative route to the destination based on that information, avoiding the prohibited areas.

[1181] In addition, based on feedback from the emotion engine, the system adjusts the travel speed and route according to the user's stress level and emotional state. For example, if the user is feeling stressed, the system will slow down the travel speed.

[1182] User (facility administrator) actions

[1183] Users manage and monitor the system.

[1184] Hardware and Software

[1185] Users use devices (e.g., PCs, tablets, smartphones) that can access the management dashboard. The dashboard uses an internet browser and is designed with an emphasis on real-time performance.

[1186] Data processing and calculation

[1187] Users can monitor the current location and operational status of their autonomous mobile devices, as well as the status of restricted area settings, in real time from the management dashboard. If necessary, they can configure specific areas to be restricted from entry from the dashboard and send that information to the server, where the settings are reflected throughout the system.

[1188] The user also confirms that the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the route and operation schedule are adjusted. Furthermore, data from the emotion engine is monitored, and system settings are adjusted according to the user's emotional state. For example, if the user's stress level is high, the robot's operation speed may be slowed down.

[1189] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1190] Server Processing Steps

[1191] Step 1: Receiving camera images

[1192] The server receives image data periodically sent from the autonomous mobile device via an HTTP POST request. This image data is temporarily saved in a buffer file called " / tmp / captured_image.jpg."

[1193] Input: Image data sent from the autonomous mobile device (e.g., / tmp / captured_image.jpg)

[1194] Output: Temporarily saved image data

[1195] Step 2: Generate the prompt

[1196] The server generates a prompt using the received image data, for example adding the text "Is this area off-limits?" along with the path of the received image.

[1197] Input: Received image data (e.g., / tmp / captured_image.jpg)

[1198] Output: Generated prompt text (e.g. "Image: / tmp / captured_image.jpg Question: Is this area off-limits?")

[1199] Step 3: Send to generative AI model

[1200] The server sends the generated prompt to the API endpoint of the generative AI model via an HTTP POST request and waits for feedback from the generative AI model.

[1201] Input: Generated prompt text

[1202] Output: Feedback from the generative AI model

[1203] Step 4: Analyze the feedback

[1204] The server analyzes the feedback received from the generative AI model. For example, if it receives feedback such as "This area is off-limits," it analyzes the content and determines whether it is a off-limits area.

[1205] Input: Feedback from the generative AI model

[1206] Output: No entry area determination result

[1207] Step 5: Map out the no-go areas

[1208] If the server determines that an area is a no-entry area, it calculates the spatial coordinates of that area. It uses a depth camera to link 3D point cloud data with 2D image data to perform detailed mapping. It then updates the map data of the autonomous mobile device.

[1209] Input: No-entry area determination results, 3D point cloud data, 2D image data

[1210] Output: Updated map data

[1211] Step 6: Managing information retention period

[1212] The server manages information about restricted areas based on a retention period. It registers the information in the database based on the retention period information from the generative AI model, and deletes or updates it after the specified period has expired.

[1213] Input: No entry area information, retention period information

[1214] Output: Database update status

[1215] Step 7: Emotion Recognition with the Emotion Engine

[1216] The server uses an emotion engine to analyze the user's speech and facial expressions to understand their emotional state, and adjusts system settings accordingly. For example, if the user is dissatisfied, the server revises the settings for no-entry areas.

[1217] Input: User speech and facial expression data

[1218] Output: System settings adjustment status

[1219] Processing steps of an autonomous mobile device

[1220] Step 1: Capture a camera image

[1221] The autonomous mobile device captures camera images at regular intervals, and the image data is temporarily stored in memory.

[1222] Input: Visual information in the environment

[1223] Output: Captured image data

[1224] Step 2: Sending images

[1225] The autonomous mobile device sends the stored images to the server. It has a retry mechanism until the transmission is successful, ensuring communication stability. It uses HTTP POST requests.

[1226] Input: Captured image data

[1227] Output: Image data sent to the server

[1228] Step 3: Receiving instructions

[1229] The autonomous mobile device receives feedback from the server and acquires information about no-entry areas. If the feedback is "This area is no-entry," the device performs processing based on that information.

[1230] Input: Feedback from the server

[1231] Output: No entry area information

[1232] Step 4: Take evasive action

[1233] The autonomous mobile device calculates a new route based on the information about the restricted area and continues moving toward the destination. It selects a different route to avoid the restricted area.

[1234] Input: No entry area information

[1235] Output: Updated route information as a result of taking evasive action

[1236] Step 5: Receiving Emotional Feedback

[1237] The autonomous mobile device receives feedback from the emotion engine and adjusts its travel speed and route according to the user's stress level and emotional state, for example, slowing down if the user is feeling stressed.

[1238] Input: Feedback from the emotion engine

[1239] Output: Adjusted speed and route information

[1240] User processing steps

[1241] Step 1: Monitor your system

[1242] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1243] Input: Various operation information, system setting information

[1244] Output: Displayed operation status and setting status

[1245] Step 2: Change the settings

[1246] Users can enter settings to restrict access to specific areas through the dashboard and send them to the server, where the information is processed and reflected throughout the system.

[1247] Input: Information about the settings you want to change

[1248] Output: Settings reflected on the server

[1249] Step 3: Check operation

[1250] The user confirms that the autonomous mobile device properly recognizes the restricted area and takes evasive action, adjusting the route and operation schedule as necessary.

[1251] Input: Operation status information of autonomous mobile device

[1252] Output: Confirmed operational status, adjusted routes and schedules

[1253] Step 4: Monitoring and Configuring Emotion Data

[1254] Users monitor data from the emotion engine and adjust system settings based on their emotional state. For example, if a user's stress level is high, the robot's speed may be slowed down.

[1255] Input: Emotion engine data

[1256] Output: Adjusted system settings

[1257] (Application example 2)

[1258] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1259] Conventional autonomous mobile devices have had difficulty in properly recognizing and avoiding no-entry areas. Furthermore, they lacked the means to dynamically manage no-entry area information and adjust their operating speed and route based on the emotional state of the user (delivery worker or customer). This has made it difficult to operate efficiently and safely in delivery operations, resulting in a decline in user satisfaction.

[1260] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for managing information about no-entry areas based on a retention period, means for supporting delivery robot operation management on a smart device application, means for calculating an alternative route to avoid the no-entry area and instructing the delivery robot to take that route, and means for analyzing the user's emotional state and adjusting the operation speed and route based on that information. This makes it possible to dynamically manage no-entry areas and realize flexible operation adjustments according to the user's emotional state.

[1261] An "autonomous mobile device" is a device that moves autonomously and performs designated tasks.

[1262] An "imaging device" is a device that includes a camera and a sensor for acquiring image data.

[1263] A "prompt" is information containing questions or instructions about image data to be input to a generative AI model.

[1264] A "generative artificial intelligence model" is an AI model that generates no-entry area determinations and other feedback based on input prompts.

[1265] "Feedback" refers to the judgment results or responses obtained from a generative AI model.

[1266] "Mapping" is the act of placing no-entry areas on a map as spatial information based on the acquired data.

[1267] The "retention period" is the period during which information regarding no-entry areas is valid, and once this period has elapsed, the information will be updated or deleted.

[1268] A "delivery robot" is a robot that automatically makes deliveries to people, vehicles, etc.

[1269] "Smart devices" are devices with advanced computing power and communication functions, and primarily refer to smartphones and tablets.

[1270] An "application" is a program or software that runs on a smart device.

[1271] An "alternate route" is a newly calculated route to avoid a no-entry area.

[1272] "Emotional state" refers to the user's emotional or psychological state, primarily stress level and satisfaction.

[1273] "Operation speed" refers to the speed at which the delivery robot moves.

[1274] A "route" is the path a delivery robot travels.

[1275] "Analysis" is the process of interpreting and making sense of data and information.

[1276] This invention relates to a system that enables autonomous mobile devices to autonomously determine no-entry areas and take evasive action. In addition, by combining this with an emotion engine that recognizes the user's emotions, it realizes more flexible and user-friendly operation management.

[1277] Server Operation

[1278] The server recognizes and maps no-entry areas using the following hardware and software:

[1279] 1. Receiving camera images

[1280] The server receives camera images periodically transmitted from the autonomous mobile device, and temporarily stores the received image data in a buffer.

[1281] 2. Prompt Generation

[1282] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[1283] Example prompt sentence:

[1284] Image: / path / to / image.jpg

[1285] Q: Is this area off-limits?

[1286] 3. Send to generative AI model

[1287] The server sends the generated prompt to the generative artificial intelligence model and waits for feedback, which includes information about the no-go area.

[1288] 4. Feedback Analysis

[1289] The feedback obtained from the generative AI model is analyzed to determine no-entry areas. The generative AI model is used for image analysis.

[1290] 5. Mapping No-Go Areas

[1291] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device, closely linking 2D image data with 3D point cloud data.

[1292] 6. Information Retention Period Management

[1293] No-entry area information is retained for a set period based on feedback from the generative AI model, after which it is deleted or updated from the database.

[1294] 7. Emotion Recognition with Emotion Engine

[1295] The emotion engine analyzes the user's speech and facial expressions and adjusts operation management based on that information.

[1296] Terminal (autonomous mobile device) operation

[1297] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action, and also adjusts its operation status based on feedback from the emotion engine.

[1298] 1. Camera image capture

[1299] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[1300] 2. Sending images

[1301] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[1302] 3. Receiving Instructions

[1303] Receive feedback from the server and obtain information about restricted areas.

[1304] 4. Taking evasive action

[1305] Based on the information about the no-entry areas, the autonomous mobile device recalculates an alternative route to the destination and avoids the no-entry areas.

[1306] 5. Receiving emotional feedback

[1307] It receives feedback from an emotion engine and adjusts its speed and route based on the user's stress level and emotional state.

[1308] User (delivery person or customer) behavior

[1309] The user manages and monitors the system.

[1310] 1. System Monitoring

[1311] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zones, in real time.

[1312] 2. Change settings

[1313] Users enter settings to prohibit entry to specific areas from the dashboard and send them to the server.

[1314] 3. Check operation

[1315] Users can check whether their autonomous mobile device is properly recognizing no-entry areas and taking evasive action, and adjust their route or operation schedule as necessary.

[1316] 4. Monitoring and setting emotion data

[1317] Users can monitor data from the emotion engine and adjust system settings based on their emotional state, for example slowing down the delivery robot if their stress levels are high.

[1318] As described above, the present invention provides means for making the operation management of autonomous mobile devices more efficient and safe.

[1319] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1320] Step 1:

[1321] The autonomous mobile device (terminal) captures camera images at regular intervals. The captured image data is temporarily stored in memory. The input is image data from the camera sensor, and the output is an image file in memory.

[1322] Step 2:

[1323] The device sends the image stored in memory to the server using an HTTP POST request. The input here is the image file in memory, and the output is the image data sent to the server. To ensure communication stability, a retry mechanism is included until the transmission is successful.

[1324] Step 3:

[1325] The server receives camera images sent from the autonomous mobile device and temporarily stores them in a buffer. The input is the image data from the device, and the output is the image data in the buffer.

[1326] Step 4:

[1327] The server generates a prompt to analyze the received image data. The prompt contains the image along with the text "Is this area off-limits?". The input is the image data in the buffer and a predefined question, and the output is the generated prompt.

[1328] Step 5:

[1329] The server sends the generated prompt to the AI ​​model and waits for feedback. The input is the prompt sentence, and the output is feedback about the no-entry area.

[1330] Step 6:

[1331] The server analyzes the feedback obtained from the generative AI model and determines the no-entry areas. The input is the feedback data returned from the generative AI model, and the output is the no-entry area determination result.

[1332] Step 7:

[1333] If a no-entry area is determined, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. The inputs are the no-entry area determination result, 2D image data, and 3D point cloud data, and the output is the updated map data.

[1334] Step 8:

[1335] The server manages information about restricted areas based on a retention period. The input is the information about restricted areas and the retention period setting, and the output is the data to be updated or deleted after the retention period has passed.

[1336] Step 9:

[1337] The server uses an emotion engine to analyze the user's speech and facial expressions, and adjusts operation management based on that information. The input is the user's speech data and facial expression data, and the output is analyzed emotional information. Based on this emotional information, the operation speed and route are adjusted.

[1338] Step 10:

[1339] The device receives feedback from the server and recalculates an alternative route based on the information on no-entry areas. The input is the no-entry area information from the server, and the output is the newly calculated alternative route.

[1340] Step 11:

[1341] The device continues moving to the destination according to the recalculated alternative route. The input is the new route information, and the output is the result of the movement.

[1342] Step 12:

[1343] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device in real time. The input is real-time data stored on the server, and the output is the operation status displayed on the user's screen.

[1344] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1345] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1346] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1347] [Third embodiment]

[1348] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1349] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1350] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1351] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1352] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1353] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1354] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1355] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1356] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1357] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1358] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1359] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1360] The present invention relates to a system for an autonomous mobile device to autonomously determine a no-entry area and take action to avoid it. The elements required to implement the present invention and their operations will be described below in detail.

[1361] Server Operation

[1362] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. The specific operation of the server is described below.

[1363] 1. Receiving camera images

[1364] The server receives camera images periodically transmitted from the autonomous mobile device, and the received image data is temporarily stored in a buffer.

[1365] 2. Prompt Generation

[1366] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[1367] 3. Send to generative AI model

[1368] The server sends the generated prompt to the artificial intelligence model and waits for a response, which includes information about the no-entry area.

[1369] 4. Feedback Analysis

[1370] The feedback from the generative artificial intelligence model is analyzed to determine whether the area is a no-entry area.

[1371] 5. Mapping No-Go Areas

[1372] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. A depth camera is used to link 3D point cloud data with 2D images to perform detailed mapping.

[1373] 6. Information Retention Period Management

[1374] Information about restricted areas is retained for a set period based on feedback from the generative AI model, and is then deleted or updated from the database after the period expires.

[1375] Terminal (autonomous mobile device) operation

[1376] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. The specific operation of the autonomous mobile device is described below.

[1377] 1. Camera image capture

[1378] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[1379] 2. Sending images

[1380] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[1381] 3. Receiving Instructions

[1382] Receive feedback from the server and obtain information about restricted areas.

[1383] 4. Taking evasive action

[1384] Based on the information on the restricted area, the autonomous mobile device recalculates an alternative route to the destination and avoids the restricted area. If it is unable to plot a route to the destination, it notifies the administrator.

[1385] User (facility administrator) actions

[1386] The user manages and monitors the system. The specific actions of the user are described below.

[1387] 1. System Monitoring

[1388] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1389] 2. Change settings

[1390] Users can enter settings to prohibit entry to specific areas from the dashboard and send them to the server. Information on the designated area and period is sent to the server and reflected throughout the system.

[1391] 3. Check operation

[1392] The user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route and operation schedule.

[1393] Specific examples

[1394] The present invention will be described below with reference to specific examples.

[1395] Server example

[1396] 1. Receiving camera images

[1397] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[1398] 2. Prompt Generation

[1399] Generate a prompt containing the received image and format it in JSON format.

[1400] json

[1401] {

[1402] "image": " / tmp / captured_image.jpg",

[1403] "question": "Is this area off-limits?"

[1404] }

[1405] 3. Sending to the generative AI model and receiving feedback

[1406] This prompt is sent to the API endpoint of the generative AI model, and feedback is received saying, "This area is off-limits."

[1407] 4. Mapping No-Go Areas

[1408] The location of the no entry sign is converted into 3D coordinates and the map data of the autonomous mobile device is updated.

[1409] 5. Information Retention Period Management

[1410] The retention period information from the generative AI model, "This information is valid for one day," is registered in the database, and a deletion task is scheduled for one day later.

[1411] Examples of autonomous mobile devices

[1412] 1. Camera image capture

[1413] The autonomous mobile device uses a camera module to capture images within the facility.

[1414] 2. Sending images

[1415] Send the image data to the server using an HTTP POST request.

[1416] 3. Receiving Instructions

[1417] Receive feedback from the server that the area is off-limits and analyze the content.

[1418] 4. Taking evasive action

[1419] Based on information about restricted areas, a new route is calculated and travel to the destination is continued.

[1420] User Examples

[1421] 1. System Monitoring

[1422] Log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[1423] 2. Change settings

[1424] Settings are made to temporarily prohibit entry to specific areas and sent to the server.

[1425] 3. Check operation

[1426] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[1427] The processing flow will be explained below.

[1428] Server Processing Steps

[1429] Step 1:

[1430] The server waits for and receives camera images sent by the autonomous mobile device via HTTP requests, and temporarily stores the received image data in a buffer.

[1431] Step 2:

[1432] The server passes the received camera image to the prompt generation module, which generates a prompt containing the image and the text "Is this area off-limits?". The prompt is then formatted into JSON format to be sent to the generative AI model.

[1433] Step 3:

[1434] The server sends the formatted prompt via an HTTPS request to the API endpoint of the generative AI model and waits for feedback from the generative AI model.

[1435] Step 4:

[1436] The server receives feedback from the generative AI model in JSON format, including information such as "no entry areas," and analyzes the content to determine whether the area is a no entry area.

[1437] Step 5:

[1438] The server references spatial data linked to the camera's viewpoint to obtain coordinate information for the restricted area based on the feedback. It uses a depth camera or stereo method to link 3D data with 2D images and calculates detailed spatial coordinates.

[1439] Step 6:

[1440] The server adds the calculated spatial coordinates to the robot's map data and maps it as a new no-entry area.

[1441] Step 7:

[1442] The server stores the retention period information provided in the feedback of the generative AI model in a database, manages the retention period of the no-entry area information, and schedules a task to update the information when the retention period expires.

[1443] Processing steps of terminal (autonomous mobile device)

[1444] Step 1:

[1445] The autonomous mobile device captures images from the camera module at regular intervals and temporarily stores them in memory.

[1446] Step 2:

[1447] The autonomous mobile device sends the saved image to a specified server endpoint via an HTTP POST request. It also has a retry mechanism until the transmission is successful, ensuring communication stability.

[1448] Step 3:

[1449] The autonomous mobile device receives feedback from the server, analyzes the content, and obtains information about restricted areas.

[1450] Step 4:

[1451] The autonomous mobile device uses the received coordinate information of the no-entry area to autonomously calculate a new route and avoid the no-entry area.

[1452] Step 5:

[1453] The autonomous mobile device instructs the navigation module to find an avoidance route and continues moving to the destination. If the device cannot find a route to the destination, it notifies the administrator.

[1454] User (facility manager) processing steps

[1455] Step 1:

[1456] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1457] Step 2:

[1458] Users can set specific areas to be off-limits as needed by entering the area's coordinates and the off-limits period into the form on the management screen and pressing the "Set" button to send the information to the server.

[1459] Step 3:

[1460] The user checks whether the autonomous mobile device is properly avoiding restricted areas and adjusts the robot's route and operation schedule as necessary.

[1461] Example 1

[1462] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1463] Autonomous mobile devices are required to accurately recognize no-entry areas and move around safely and efficiently. However, with conventional technology, the determination of no-entry areas and updating of map data are often done manually, making it difficult to respond in real time. In addition, managing the retention period for information about no-entry areas and taking appropriate evasive action are difficult, which can lead to reduced operational efficiency and safety for autonomous mobile devices.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1465] In this invention, the server includes means for transmitting image data acquired by an imaging device mounted on the autonomous mobile device, means for generating the image data as a prompt, means for transmitting the prompt to a generative artificial intelligence model and receiving feedback regarding no-entry areas, means for analyzing the feedback and reflecting the no-entry areas on a map, means for managing information regarding the no-entry areas based on a retention period, and means for recalculating the route of the autonomous mobile device to avoid the no-entry areas. This enables the autonomous mobile device to autonomously recognize no-entry areas, update map data in real time, and take efficient avoidance actions.

[1466] An "autonomous mobile device" is a device equipped with sensors, cameras, and various control programs that moves and operates on its own judgment.

[1467] An "imaging device" is hardware for capturing images or video, such as a camera or depth sensor.

[1468] "Image data" is data that digitally represents visual information acquired by an imaging device.

[1469] A "prompt" is a data structure containing instructions or questions that are input to a generative AI model.

[1470] A "generative artificial intelligence model" is an AI system that uses machine learning and deep learning technologies to analyze and make predictions based on input data.

[1471] "Feedback" refers to the analysis results and response data returned as a response from the generative AI model.

[1472] A "no-entry area" is a specific spatial region to which access is restricted for the purpose of managing the safety of people and equipment and improving work efficiency.

[1473] "Reflecting on the map" means updating geographical data based on the acquired information and accurately showing the new information.

[1474] "Retention period" refers to the period for which information is considered valid, and means maintaining and managing the information until that period has elapsed.

[1475] "Recalculating a route" refers to recalculating the optimal route between the current location and the destination.

[1476] "3D point cloud data" is a collection of a series of points that make up a space, and each point is expressed in a 3D coordinate system (x, y, z).

[1477] The present invention relates to a system for an autonomous mobile device to autonomously determine a no-entry area and take action to avoid the area. Implementing the present invention requires the following specific steps.

[1478] Server Operation

[1479] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. Specific hardware used is a camera module (image capture device) and a depth sensor (a device for acquiring 3D point cloud data).

[1480] First, the server periodically receives camera images sent from the autonomous mobile device. This image data is temporarily stored in a buffer. The reception process is often performed using an HTTP POST request.

[1481] Next, the server generates a prompt to analyze the received image data. This prompt contains the path to the image file and the text "Is this area off-limits?". For example, the following prompt is generated:

[1482] Image: / tmp / captured_image.jpg

[1483] Q: Is this area off-limits?

[1484] The generated prompt is sent to the API endpoint of the generative AI model. After being sent to the API endpoint, the server waits for feedback from the generative AI model. The feedback includes information on whether the area is a no-entry area.

[1485] The server receives feedback from the AI ​​model and analyzes its contents. For example, it may receive feedback such as "This area is off-limits." If this feedback is analyzed and it is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. Specifically, it links the 3D point cloud data acquired from the depth camera with 2D images to perform detailed mapping.

[1486] Finally, the server manages the information about restricted areas based on a retention period. For example, if the generative AI model instructs the server to keep the information for one day, the server will register the information in the database and schedule a deletion task one day later.

[1487] Terminal (autonomous mobile device) operation

[1488] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. Specifically, it performs the following operations.

[1489] First, the autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in the device's memory.

[1490] Next, the autonomous mobile device transmits the stored image data to the server. If the transmission is unsuccessful, a retry mechanism is used to attempt retransmission.

[1491] Receives feedback from the server and obtains information on restricted areas. Analyzes the information, recalculates the route based on the restricted area information, and creates an alternative route to the destination. If no suitable route is found, notifies the administrator.

[1492] User (facility administrator) actions

[1493] Users use the management dashboard to monitor and manage the entire system. The specific actions of users are as follows:

[1494] First, the user accesses the management dashboard and monitors the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1495] Next, the user enters settings for prohibiting entry to specific areas on the dashboard and sends them to the server. Information on the specified areas and period is sent to the server and reflected throughout the system.

[1496] Finally, the user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route or operation schedule.

[1497] Specific examples

[1498] For example, the following scenario is possible:

[1499] The server receives the image data from the "Autonomous Operation Sensor" and saves it as / tmp / captured_image.jpg. It generates a prompt containing the received image and formats it as follows:

[1500] Image: / tmp / captured_image.jpg

[1501] Q: Is this area off-limits?

[1502] This prompt is sent to the API endpoint of the generative AI model, and feedback is received stating, "This area is prohibited from entry." The location of the no-entry sign is converted into 3D coordinates, and the map data of the autonomous mobile device is updated. The generative AI model also registers the retention period information, "This information is valid for one day," in the database, and schedules a deletion task one day later.

[1503] The autonomous mobile device captures images using a camera module within the facility and sends them to the server via an HTTP POST request. It then receives feedback from the server, calculates a new route based on the information on "no entry areas," and continues moving toward its destination.

[1504] Users log in to the dashboard, monitor the operation status of the autonomous mobile device in real time, set up temporary prohibitions on certain areas, and send the information to the server. The system checks whether the robot is avoiding the prohibited areas and adjusts its route and schedule as necessary.

[1505] The above describes a specific embodiment of the invention. This system enables an autonomous mobile device to autonomously recognize no-entry areas, update map data in real time, and take efficient avoidance actions.

[1506] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1507] Server Operation

[1508] Step 1:

[1509] The server receives camera images from the autonomous mobile device.

[1510] Specific behavior:

[1511] The server receives the HTTP POST request and acquires the camera image data. The received image is temporarily stored in buffer memory.

[1512] Input: Camera image data sent from an autonomous mobile device.

[1513] Output: The image data will be saved in / tmp / captured_image.jpg.

[1514] Step 2:

[1515] The server generates a prompt to analyze the received image data.

[1516] Specific behavior:

[1517] The server generates a JSON prompt containing the file path of the received image and the text "Is this area off-limits?"

[1518] Input: The camera image saved as / tmp / captured_image.jpg and the question text.

[1519] Output: The generated prompt (e.g., {"image": " / tmp / captured_image.jpg", "question": "Is this area off-limits?"}).

[1520] Step 3:

[1521] The server sends the generated prompts to the generating artificial intelligence model.

[1522] Specific behavior:

[1523] The server sends an HTTP POST request to the API endpoint of the generated AI model.

[1524] Input: The generated prompt.

[1525] Output: The request sent to the API endpoint of the generative AI model.

[1526] Step 4:

[1527] The server receives and analyzes feedback from the generative AI model.

[1528] Specific behavior:

[1529] The server receives the HTTP response from the generated AI model and analyzes the feedback content, which is "No entry."

[1530] Input: Feedback from the generative AI model.

[1531] Output: No entry area determination result.

[1532] Step 5:

[1533] If the server determines that the area is a no-entry area, it calculates the spatial coordinates of the area and updates the map data.

[1534] Specific behavior:

[1535] The 3D point cloud data acquired by the depth camera is linked to the 2D images to calculate the precise location of no-entry areas and update the map data.

[1536] Input: No-entry area determination results, 3D point cloud data, and 2D image data.

[1537] Output: Updated map data.

[1538] Step 6:

[1539] The server manages the information about the no-entry areas based on a retention period.

[1540] Specific behavior:

[1541] Receive retention period information (e.g., "valid for 1 day") from the generated AI model and register it in the database. Schedule a deletion task after the specified period.

[1542] Input: No entry area information and retention period information.

[1543] Output: No entry area information registered in the database with retention period.

[1544] Terminal (autonomous mobile device) operation

[1545] Step 1:

[1546] The autonomous mobile device captures camera images at regular intervals.

[1547] Specific behavior:

[1548] The autonomous mobile device periodically activates the camera module to capture image data and temporarily store it in memory.

[1549] Input: A periodic timer event.

[1550] Output: Captured camera image data.

[1551] Step 2:

[1552] The autonomous mobile device transmits the captured images to a server.

[1553] Specific behavior:

[1554] The autonomous mobile device sends the stored image data to the server using an HTTP POST request. If the transmission is unsuccessful, the device attempts to retransmit the data using a retry mechanism.

[1555] Input: Captured camera image data.

[1556] Output: Image data sent to the server.

[1557] Step 3:

[1558] The autonomous mobile device receives feedback from the server.

[1559] Specific behavior:

[1560] Wait for feedback sent from the server and analyze it as soon as it is received.

[1561] Input: Feedback data from the server.

[1562] Output: Parsed feedback data.

[1563] Step 4:

[1564] The autonomous mobile device takes evasive action based on information about no-entry areas.

[1565] Specific behavior:

[1566] The system recalculates routes based on information about restricted areas and plans alternative routes to the destination. If no suitable route is found, it notifies the administrator.

[1567] Input: Parsed no-go area information.

[1568] Output: Recalculated route information or notification to administrator.

[1569] User (facility administrator) actions

[1570] Step 1:

[1571] Users access an administrative dashboard to monitor the status of their systems.

[1572] Specific behavior:

[1573] Through the dashboard, users can check the current location of the autonomous mobile device, its operating status, and the status of no-entry areas in real time.

[1574] Input: Dashboard access information and system status data.

[1575] Output: Visualized information showing the current state.

[1576] Step 2:

[1577] The user inputs settings to prohibit entry into specific areas.

[1578] Specific behavior:

[1579] The user designates a specific area as off-limits on the dashboard and sends that information to the server.

[1580] Input: User configuration input.

[1581] Output: No entry configuration information sent to the server.

[1582] Step 3:

[1583] The user checks the operating status of the autonomous mobile device.

[1584] Specific behavior:

[1585] The user monitors whether the autonomous mobile device recognizes no-entry areas and takes appropriate avoidance actions, adjusting the route or schedule as necessary.

[1586] Input: Dashboard traffic data.

[1587] Output: Check results and any necessary adjustments.

[1588] As described above, by providing detailed explanations for each processing step, including specific operations and inputs and outputs, we aim to ensure that the operation of this system is accurate and efficient.

[1589] (Application example 1)

[1590] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1591] Conventional autonomous mobile devices have limited capabilities for detecting and avoiding restricted areas, which has led to problems in ensuring safety, particularly in complex industrial facilities. There is also a risk that workers may mistakenly enter restricted or dangerous areas, which could lead to workplace accidents. Therefore, there is a need to improve safety by improving the ability of autonomous mobile devices to recognize restricted areas and by adding a real-time warning function to workers.

[1592] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1593] In this invention, the server includes means for periodically receiving camera images and temporarily storing them in a buffer, means for generating prompts for analyzing the received image data, means for sending the generated prompts to a generative artificial intelligence model and receiving feedback regarding no-entry areas, means for analyzing the feedback and mapping the no-entry areas, means for managing information regarding no-entry areas based on a retention period, and means for notifying smart devices worn by workers in industrial facilities in real time of information regarding no-entry areas and dangerous areas. This not only enables autonomous mobile devices to more accurately recognize and avoid no-entry areas, but also allows workers to be immediately warned of no-entry areas and dangerous areas, significantly improving safety.

[1594] An "autonomous mobile device" is a mechanical device that uses sensors to recognize its surrounding environment, automatically plans a route to its destination, and moves accordingly.

[1595] "Camera image" refers to visual information acquired by a camera mounted on an autonomous mobile device.

[1596] A "prompt" is a question generated based on camera images, and is text information provided to a generative artificial intelligence model to determine no-entry areas.

[1597] A "generative artificial intelligence model" is a system with machine learning algorithms used to generate answers to prompts.

[1598] "Feedback" is a response output from the generative artificial intelligence model, and includes information about no-entry areas.

[1599] "Mapping" is the process of converting the location of the no-entry area into spatial coordinates and recording them as map data for the autonomous mobile device.

[1600] "Retention period" refers to the period for which information regarding restricted areas is stored within the system.

[1601] "Industrial facility" refers to buildings and premises where manufacturing or other industrial activity is carried out.

[1602] "Smart equipment" refers to wearable devices worn by factory workers that have the ability to receive and display information about restricted and dangerous areas in real time.

[1603] "Real-time notification" means instantly transmitting information about restricted or dangerous areas to users.

[1604] A "hazardous area" is an area that poses a potential hazard to personnel or equipment.

[1605] The present invention relates to a system that enables autonomous mobile devices to autonomously determine restricted areas and take evasive action. In particular, the system can improve worker safety by linking with smart devices worn by workers in industrial facilities and notifying them of restricted or dangerous areas in real time.

[1606] Server Operation

[1607] The server receives camera images sent from the autonomous mobile device, analyzes them, and determines which areas are off-limits. The server uses the following hardware and software:

[1608] Hardware: high performance servers, storage devices, network interfaces

[1609] Software: image analysis algorithms, generative AI models, databases

[1610] The server first periodically receives camera images sent from the autonomous mobile device and stores them in a buffer. Next, it generates a prompt to analyze the received image data. This prompt includes the image data and the question, "Is this area off-limits?" The generated prompt is sent to the generative AI model, which receives feedback regarding the off-limits area. The received feedback is analyzed, and if it is determined to be an off-limits area, the location is converted into spatial coordinates and mapped. Finally, it sets a retention period for information regarding off-limits areas and manages them in a database.

[1611] Terminal (autonomous mobile device) operation

[1612] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. The hardware and software of the terminal are as follows.

[1613] Hardware: Camera module, communication module, control unit

[1614] Software: Image capture software, communication protocols, path planning algorithms

[1615] The autonomous mobile device first captures images at regular intervals using a camera module. The captured image data is stored in memory and sent to a server. After receiving feedback from the server and obtaining information about no-entry areas, the device calculates an alternative route to the destination and continues moving while avoiding the no-entry areas.

[1616] User (factory workers and managers) actions

[1617] Factory workers and managers manage and monitor the system. The hardware and software used are as follows:

[1618] Hardware: Smart helmet, dashboard display

[1619] Software: Real-time monitoring application, notification system, configuration management interface

[1620] Factory workers wear smart helmets and receive real-time notifications of restricted or dangerous areas. For example, a message such as "This area is prohibited. Please avoid it" appears on the helmet's display. Managers can also use a dashboard to monitor the operation of the autonomous mobile devices and adjust zone settings and schedules.

[1621] Specific examples

[1622] Specific examples are shown below.

[1623] The server periodically receives and analyzes "image data within the facility."

[1624] An example of a prompt statement is: "Is this area off-limits?"

[1625] Workers can take immediate action when the message "Danger area. Please evacuate" appears on the display inside the smart helmet.

[1626] This provides a system that not only enables autonomous mobile devices to accurately recognize and avoid no-entry areas, but also improves the safety of factory workers.

[1627] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1628] Step 1:

[1629] The server periodically receives image data from the autonomous mobile device. The input is camera image data, which is temporarily stored in the server's buffer. This buffering allows for smooth subsequent data analysis.

[1630] Step 2:

[1631] The server generates a prompt based on the received image data. The input is the image data saved in the previous step, and the output is the prompt text. The server generates a prompt in JSON format that includes the image data and the text "Is this area off-limits?"

[1632] Step 3:

[1633] The server sends the generated prompt to the generative AI model and receives feedback. The input is the generated prompt, and the output is feedback from the generative AI model. The server determines whether the area is off-limits based on this feedback.

[1634] Step 4:

[1635] The server analyzes the feedback and maps the no-entry areas to spatial coordinates. The input is the feedback data from the generative AI model, and the output is the spatial coordinate data of the no-entry areas. The server links the 2D image data with the 3D point cloud data to obtain detailed location information.

[1636] Step 5:

[1637] The server manages information about restricted areas based on retention periods. The input is the spatial coordinate data of the restricted areas and retention period information, and the output is the updated database state. The server sets a schedule for deleting or updating the data after the set period.

[1638] Step 6:

[1639] The autonomous mobile device captures camera images at regular intervals. The input is real-time video data, and the output is captured still images. The images are stored in memory via the camera module.

[1640] Step 7:

[1641] The autonomous mobile device sends the captured image to the server. The input is the captured image data, and the output is an HTTP request to the server. The image data is sent to the server using the communication module.

[1642] Step 8:

[1643] The autonomous mobile device receives feedback from the server and acquires the no-entry area information. The input is the feedback data sent from the server, and the output is the analyzed no-entry area information. The information is acquired through the communication module.

[1644] Step 9:

[1645] The autonomous mobile device calculates a new route based on the no-entry area information and continues moving toward the destination. The input is the no-entry area information and the output is a new navigation route. A path planning algorithm is used to calculate the avoidance route.

[1646] Step 10:

[1647] Users wear smart devices and receive real-time notifications of restricted or dangerous areas. The input is a warning message from the server, and the output is the warning information displayed on the smart device's display. Users can immediately check this information and take evasive action.

[1648] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1649] The present invention relates to a system that enables an autonomous mobile device to autonomously determine no-entry areas and take evasive action. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it achieves more flexible and user-friendly operation management. Below, specific embodiments of each element and its operation for implementing the present invention will be described.

[1650] Server Operation

[1651] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. It also acquires user emotional information and adjusts the settings of the entire system. The specific operation of the server is described below.

[1652] 1. Receiving camera images

[1653] The server receives camera images periodically transmitted from the autonomous mobile device, and the received image data is temporarily stored in a buffer.

[1654] 2. Prompt Generation

[1655] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[1656] 3. Send to generative AI model

[1657] The server sends the generated prompt to the generative AI model and waits for feedback, which includes information about the no-go area.

[1658] 4. Feedback Analysis

[1659] The feedback from the generative artificial intelligence model is analyzed to determine whether the area is off-limits.

[1660] 5. Mapping No-Go Areas

[1661] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. A depth camera is used to link 3D point cloud data with 2D images to perform detailed mapping.

[1662] 6. Information Retention Period Management

[1663] Information about restricted areas is retained for a set period based on feedback from the generative AI model, and is then deleted or updated from the database after the period expires.

[1664] 7. Emotion Recognition with Emotion Engine

[1665] To obtain the user's emotional information, the emotion engine analyzes the user's speech and facial expressions, and adjusts the no-entry zone settings accordingly.

[1666] Terminal (autonomous mobile device) operation

[1667] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action. It also adjusts its operation status based on feedback from the emotion engine. The specific operation of the autonomous mobile device is described below.

[1668] 1. Camera image capture

[1669] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[1670] 2. Sending images

[1671] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[1672] 3. Receiving Instructions

[1673] Receive feedback from the server and obtain information about restricted areas.

[1674] 4. Taking evasive action

[1675] Based on the information about the no-entry areas, the autonomous mobile device recalculates an alternative route to the destination and avoids the no-entry areas.

[1676] 5. Receiving emotional feedback

[1677] It receives feedback from an emotion engine and adjusts its speed and route based on the user's stress level and emotional state.

[1678] User (facility administrator) actions

[1679] The user manages and monitors the system. The specific actions of the user are described below.

[1680] 1. System Monitoring

[1681] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1682] 2. Change settings

[1683] Users can enter settings to prohibit entry to specific areas from the dashboard and send them to the server. Information on the designated area and period is sent to the server and reflected throughout the system.

[1684] 3. Check operation

[1685] The user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route and operation schedule.

[1686] 4. Monitoring and setting emotion data

[1687] Users can monitor data from the emotion engine and adjust system settings based on their emotional state, for example slowing down the robot's speed if stress levels are high.

[1688] Specific examples

[1689] The present invention will be described below with reference to specific examples.

[1690] Server example

[1691] 1. Receiving camera images

[1692] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[1693] 2. Generate and send prompts

[1694] Generate a prompt containing the received image and format it in JSON format.

[1695] json

[1696] {

[1697] "image": " / tmp / captured_image.jpg",

[1698] "question": "Is this area off-limits?"

[1699] }

[1700] This prompt is sent to the API endpoint of the generative AI model, and feedback is received saying, "This area is off-limits."

[1701] 3. Mapping No-Go Areas

[1702] The location of the no entry sign is converted into 3D coordinates and the map data of the autonomous mobile device is updated.

[1703] 4. Information Retention Period Management

[1704] The retention period information from the generative AI model, "This information is valid for one day," is registered in the database, and a deletion task is scheduled for one day later.

[1705] 5. Emotion Recognition by Emotion Engine

[1706] The system analyzes the user's speech and facial expressions and adjusts the settings of no-entry areas based on that information.

[1707] Examples of autonomous mobile devices

[1708] 1. Camera image capture

[1709] The autonomous mobile device uses a camera module to capture images within the facility.

[1710] 2. Sending images

[1711] Send the image data to the server using an HTTP POST request.

[1712] 3. Receiving and parsing instructions

[1713] Receive feedback from the server that the area is off-limits and analyze the content.

[1714] 4. Taking evasive action

[1715] Based on information about restricted areas, a new route is calculated and travel to the destination is continued.

[1716] 5. Receiving emotional feedback

[1717] It receives feedback from the emotion engine and adjusts its speed and route according to stress levels.

[1718] User Examples

[1719] 1. System Monitoring

[1720] Log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[1721] 2. Change settings

[1722] Settings are made to temporarily prohibit entry to specific areas and sent to the server.

[1723] 3. Check operation

[1724] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[1725] 4. Monitoring and setting emotion data

[1726] The system settings are adjusted based on emotional information obtained from the emotion engine, optimizing the robot's operation according to the user's stress level.

[1727] The processing flow will be explained below.

[1728] Server Processing Steps

[1729] Step 1:

[1730] The server waits for and receives camera images sent by the autonomous mobile device via HTTP requests, and temporarily stores the received image data in a buffer.

[1731] Step 2:

[1732] The server passes the received camera image to the prompt generation module, which generates a prompt containing the image and the text "Is this area off-limits?". The prompt is then formatted into JSON format to be sent to the generative AI model.

[1733] Step 3:

[1734] The server sends the formatted prompt via an HTTPS request to the API endpoint of the generative AI model and waits for feedback from the generative AI model.

[1735] Step 4:

[1736] The server receives feedback from the generative AI model in JSON format, including information such as "no entry areas," and analyzes the content to determine whether the area is a no entry area.

[1737] Step 5:

[1738] The server references spatial data linked to the camera's viewpoint to obtain coordinate information for the restricted area based on the feedback. It uses a depth camera or stereo method to link 3D data with 2D images and calculates detailed spatial coordinates.

[1739] Step 6:

[1740] The server adds the calculated spatial coordinates to the robot's map data and displays it on the map as a new no-entry area.

[1741] Step 7:

[1742] The server stores the retention period information provided in the feedback of the generative AI model in a database, manages the retention period of the no-entry area information, and schedules a task to update the information when the retention period expires.

[1743] Step 8:

[1744] The server uses an emotion engine to analyze the user's speech and facial expressions, and adjusts the settings for no-entry areas and other operational parameters based on the analysis results.

[1745] Processing steps of terminal (autonomous mobile device)

[1746] Step 1:

[1747] The autonomous mobile device captures images from the camera module at regular intervals and temporarily stores them in memory.

[1748] Step 2:

[1749] The autonomous mobile device sends the saved image to a specified server endpoint via an HTTP POST request. It also has a retry mechanism until the transmission is successful, ensuring communication stability.

[1750] Step 3:

[1751] The autonomous mobile device receives feedback from the server, analyzes the content, and obtains information about restricted areas.

[1752] Step 4:

[1753] The autonomous mobile device uses the received coordinate information of the no-entry area to autonomously calculate a new route and avoid the no-entry area.

[1754] Step 5:

[1755] The autonomous mobile device instructs the navigation module to find an avoidance route and continues moving to the destination. If the device cannot find a route to the destination, it notifies the administrator.

[1756] Step 6:

[1757] The autonomous mobile device receives feedback from the emotion engine and adjusts its travel speed and route based on the user's stress level and emotional state.

[1758] User (facility manager) processing steps

[1759] Step 1:

[1760] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1761] Step 2:

[1762] Users can set specific areas to be off-limits as needed by entering the area's coordinates and the off-limits period into the form on the management screen and pressing the "Set" button to send the information to the server.

[1763] Step 3:

[1764] The user checks whether the autonomous mobile device is properly avoiding restricted areas and adjusts the robot's route and operation schedule as necessary.

[1765] Step 4:

[1766] Based on the emotional information obtained from the emotion engine, the user can adjust the system settings according to the user's emotional state, for example, slowing down the robot's speed if the user's stress level is high.

[1767] As a concrete example, a specific scenario for each step will be explained below.

[1768] Server example

[1769] Step 1:

[1770] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[1771] Step 2:

[1772] The prompt generation module generates a prompt in JSON format that includes the received image and the question "Is this area off-limits?"

[1773] Step 3:

[1774] The generated prompt is sent via an HTTPS request to the API endpoint of the generated artificial intelligence model.

[1775] Step 4:

[1776] The generative AI model receives feedback such as "This area is off-limits" and parses it in JSON format.

[1777] Step 5:

[1778] Based on the feedback, the 3D coordinates of the no-entry area are calculated from the image and updated in the map data of the autonomous mobile device.

[1779] Step 6:

[1780] The calculated coordinates of the no-entry area are added to the robot's map data, and the mapping is updated.

[1781] Step 7:

[1782] Store retention period information in a database and schedule a task to update the information once the retention period has expired.

[1783] Step 8:

[1784] The emotion engine is used to analyze the user's speech and facial expressions, and the system adjusts no-entry zone settings and operating parameters based on the results.

[1785] Examples of autonomous mobile devices

[1786] Step 1:

[1787] The autonomous mobile device uses a camera module to capture images within the facility.

[1788] Step 2:

[1789] The captured image data is sent to the server via an HTTP POST request.

[1790] Step 3:

[1791] It receives feedback from the server that the area is off-limits and analyzes the content.

[1792] Step 4:

[1793] Based on information about restricted areas, the system calculates a new route and autonomously avoids restricted areas.

[1794] Step 5:

[1795] The system instructs the navigation module to find an avoidance route and continues moving to the destination. If the system is unable to find a route to the destination, it notifies the administrator.

[1796] Step 6:

[1797] It receives feedback from the emotion engine and adjusts its speed and route according to the user's stress level.

[1798] User Examples

[1799] Step 1:

[1800] Users log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[1801] Step 2:

[1802] Set up a specific area to be temporarily off-limits, enter the coordinates and period, and send it to the server.

[1803] Step 3:

[1804] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[1805] Step 4:

[1806] The system settings are adjusted based on emotional information obtained from the emotion engine, optimizing the robot's operation according to the user's stress level.

[1807] Example 2

[1808] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1809] Conventional autonomous mobile devices have difficulty in accurately identifying no-entry areas and taking evasive action. Furthermore, they lacked operational control that took into account the user's emotional state, resulting in a lack of flexibility and user-friendliness. Furthermore, they lacked management of the retention period for no-entry area information and detailed mapping of spatial coordinates.

[1810] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1811] In this invention, the server includes a means for receiving camera images, a means for generating prompts, a means for transmitting the prompts to a generative AI model and receiving feedback, a means for analyzing the feedback and mapping no-entry areas, a means for managing no-entry area information based on a retention period, and a means for acquiring emotion information using an emotion engine and adjusting system settings based on the emotion information. This enables the autonomous mobile device to appropriately determine and avoid no-entry areas, realizing flexible and user-friendly operation management based on the user's emotional state.

[1812] An "autonomous mobile device" is a device that moves autonomously without external instructions and performs designated tasks using specific programs and sensors.

[1813] A "camera" is a device that converts light into an electrical signal and records it as image data.

[1814] "Image data" is a digital representation of visual information captured by a camera.

[1815] A "prompt" is text data that includes specific image data along with questions or instructions for the generative AI model.

[1816] A "generative AI model" is a computer model that uses artificial intelligence techniques to generate feedback based on given input data (prompts).

[1817] "Feedback" is a response from a generative AI model that contains useful information or instructions for the system.

[1818] A "no-entry area" is a specific area where autonomous mobile devices are prohibited from entering.

[1819] "Mapping" is the process of locating a specific area or element on a coordinate system and representing that information visually or digitally.

[1820] "Retention Period" means the period for which collected information is stored and valid in our database.

[1821] An "emotion engine" is a technology that analyzes a user's speech and facial expressions to determine their emotional state.

[1822] "System settings" are various configurations and adjustment items that determine the operation and behavior of the entire system.

[1823] This invention is a system in which an autonomous mobile device autonomously determines no-entry areas and takes evasive action. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, flexible and user-friendly operation management is realized. Below, specific embodiments of each element and its operation for implementing this invention are described.

[1824] Server Operation

[1825] The server receives camera images from the autonomous mobile device, analyzes them, maps out no-entry areas, and acquires user emotional information to adjust the overall system settings.

[1826] Hardware and Software

[1827] The server uses a computer server equipped with a high-performance processor and large amount of memory, and the HTTP protocol is used for communication. SSDs are used for disk storage, enabling high-speed reading and writing of image data. A cloud-based service equipped with a generative AI model is used for analysis.

[1828] Data processing and calculation

[1829] The server receives camera images periodically sent from the autonomous mobile device. The received image data is temporarily stored in a buffer and a prompt is generated based on the image data. For example, a prompt such as "Is this area off-limits?" is added.

[1830] Examples of prompts:

[1831] Image: / tmp / captured_image.jpg

[1832] Q: Is this area off-limits?

[1833] This prompt is sent to the generative AI model, which receives feedback regarding the no-entry area. The feedback contains information about the no-entry area, which is then analyzed to determine whether it is a no-entry area. If it is determined to be a no-entry area, the spatial coordinates of that area are calculated and the map data of the autonomous mobile device is updated.

[1834] Furthermore, information about restricted areas is managed based on a retention period, and is deleted or updated from the database after the specified period has expired. An emotion engine is used to analyze the user's speech and facial expressions to understand the user's emotional state. Based on this, system settings such as restricted area settings and operating speeds are adjusted.

[1835] Operation of autonomous mobile device

[1836] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action, and also adjusts its operation status based on feedback from the emotion engine.

[1837] Hardware and Software

[1838] Autonomous mobile devices use mobile devices equipped with camera modules, processors, communication modules, etc. For communication, they use high-speed communication technologies such as Wi-Fi and 5G.

[1839] Data processing and calculation

[1840] The autonomous mobile device captures camera images at regular intervals and temporarily stores the image data in its memory. The stored images are then sent to the server using an HTTP POST request. After receiving feedback about prohibited areas from the server, the device recalculates an alternative route to the destination based on that information, avoiding the prohibited areas.

[1841] In addition, based on feedback from the emotion engine, the system adjusts the travel speed and route according to the user's stress level and emotional state. For example, if the user is feeling stressed, the system will slow down the travel speed.

[1842] User (facility administrator) actions

[1843] Users manage and monitor the system.

[1844] Hardware and Software

[1845] Users use devices (e.g., PCs, tablets, smartphones) that can access the management dashboard. The dashboard uses an internet browser and is designed with an emphasis on real-time performance.

[1846] Data processing and calculation

[1847] Users can monitor the current location and operational status of their autonomous mobile devices, as well as the status of restricted area settings, in real time from the management dashboard. If necessary, they can configure specific areas to be restricted from entry from the dashboard and send that information to the server, where the settings are reflected throughout the system.

[1848] The user also confirms that the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the route and operation schedule are adjusted. Furthermore, data from the emotion engine is monitored, and system settings are adjusted according to the user's emotional state. For example, if the user's stress level is high, the robot's operation speed may be slowed down.

[1849] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1850] Server Processing Steps

[1851] Step 1: Receiving camera images

[1852] The server receives image data periodically sent from the autonomous mobile device via an HTTP POST request. This image data is temporarily saved in a buffer file called " / tmp / captured_image.jpg."

[1853] Input: Image data sent from the autonomous mobile device (e.g., / tmp / captured_image.jpg)

[1854] Output: Temporarily saved image data

[1855] Step 2: Generate the prompt

[1856] The server generates a prompt using the received image data, for example adding the text "Is this area off-limits?" along with the path of the received image.

[1857] Input: Received image data (e.g., / tmp / captured_image.jpg)

[1858] Output: Generated prompt text (e.g. "Image: / tmp / captured_image.jpg Question: Is this area off-limits?")

[1859] Step 3: Send to generative AI model

[1860] The server sends the generated prompt to the API endpoint of the generative AI model via an HTTP POST request and waits for feedback from the generative AI model.

[1861] Input: Generated prompt text

[1862] Output: Feedback from the generative AI model

[1863] Step 4: Analyze the feedback

[1864] The server analyzes the feedback received from the generative AI model. For example, if it receives feedback such as "This area is off-limits," it analyzes the content and determines whether it is a off-limits area.

[1865] Input: Feedback from the generative AI model

[1866] Output: No entry area determination result

[1867] Step 5: Map out the no-go areas

[1868] If the server determines that an area is a no-entry area, it calculates the spatial coordinates of that area. It uses a depth camera to link 3D point cloud data with 2D image data to perform detailed mapping. It then updates the map data of the autonomous mobile device.

[1869] Input: No-entry area determination results, 3D point cloud data, 2D image data

[1870] Output: Updated map data

[1871] Step 6: Managing information retention period

[1872] The server manages information about restricted areas based on a retention period. It registers the information in the database based on the retention period information from the generative AI model, and deletes or updates it after the specified period has expired.

[1873] Input: No entry area information, retention period information

[1874] Output: Database update status

[1875] Step 7: Emotion Recognition with the Emotion Engine

[1876] The server uses an emotion engine to analyze the user's speech and facial expressions to understand their emotional state, and adjusts system settings accordingly. For example, if the user is dissatisfied, the server revises the settings for no-entry areas.

[1877] Input: User speech and facial expression data

[1878] Output: System settings adjustment status

[1879] Processing steps of an autonomous mobile device

[1880] Step 1: Capture a camera image

[1881] The autonomous mobile device captures camera images at regular intervals, and the image data is temporarily stored in memory.

[1882] Input: Visual information in the environment

[1883] Output: Captured image data

[1884] Step 2: Sending images

[1885] The autonomous mobile device sends the stored images to the server. It has a retry mechanism until the transmission is successful, ensuring communication stability. It uses HTTP POST requests.

[1886] Input: Captured image data

[1887] Output: Image data sent to the server

[1888] Step 3: Receiving instructions

[1889] The autonomous mobile device receives feedback from the server and acquires information about no-entry areas. If the feedback is "This area is no-entry," the device performs processing based on that information.

[1890] Input: Feedback from the server

[1891] Output: No entry area information

[1892] Step 4: Take evasive action

[1893] The autonomous mobile device calculates a new route based on the information about the restricted area and continues moving toward the destination. It selects a different route to avoid the restricted area.

[1894] Input: No entry area information

[1895] Output: Updated route information as a result of taking evasive action

[1896] Step 5: Receiving Emotional Feedback

[1897] The autonomous mobile device receives feedback from the emotion engine and adjusts its travel speed and route according to the user's stress level and emotional state, for example, slowing down if the user is feeling stressed.

[1898] Input: Feedback from the emotion engine

[1899] Output: Adjusted speed and route information

[1900] User processing steps

[1901] Step 1: Monitor your system

[1902] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[1903] Input: Various operation information, system setting information

[1904] Output: Displayed operation status and setting status

[1905] Step 2: Change the settings

[1906] Users can enter settings to restrict access to specific areas through the dashboard and send them to the server, where the information is processed and reflected throughout the system.

[1907] Input: Information about the settings you want to change

[1908] Output: Settings reflected on the server

[1909] Step 3: Check operation

[1910] The user confirms that the autonomous mobile device properly recognizes the restricted area and takes evasive action, adjusting the route and operation schedule as necessary.

[1911] Input: Operation status information of autonomous mobile device

[1912] Output: Confirmed operational status, adjusted routes and schedules

[1913] Step 4: Monitoring and Configuring Emotion Data

[1914] Users monitor data from the emotion engine and adjust system settings based on their emotional state. For example, if a user's stress level is high, the robot's speed may be slowed down.

[1915] Input: Emotion engine data

[1916] Output: Adjusted system settings

[1917] (Application example 2)

[1918] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1919] Conventional autonomous mobile devices have had difficulty in properly recognizing and avoiding no-entry areas. Furthermore, they lacked the means to dynamically manage no-entry area information and adjust their operating speed and route based on the emotional state of the user (delivery worker or customer). This has made it difficult to operate efficiently and safely in delivery operations, resulting in a decline in user satisfaction.

[1920] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for managing information about no-entry areas based on a retention period, means for supporting delivery robot operation management on a smart device application, means for calculating an alternative route to avoid the no-entry area and instructing the delivery robot to take that route, and means for analyzing the user's emotional state and adjusting the operation speed and route based on that information. This makes it possible to dynamically manage no-entry areas and realize flexible operation adjustments according to the user's emotional state.

[1921] An "autonomous mobile device" is a device that moves autonomously and performs designated tasks.

[1922] An "imaging device" is a device that includes a camera and a sensor for acquiring image data.

[1923] A "prompt" is information containing questions or instructions about image data to be input to a generative AI model.

[1924] A "generative artificial intelligence model" is an AI model that generates no-entry area determinations and other feedback based on input prompts.

[1925] "Feedback" refers to the judgment results or responses obtained from a generative AI model.

[1926] "Mapping" is the act of placing no-entry areas on a map as spatial information based on the acquired data.

[1927] The "retention period" is the period during which information regarding no-entry areas is valid, and once this period has elapsed, the information will be updated or deleted.

[1928] A "delivery robot" is a robot that automatically makes deliveries to people, vehicles, etc.

[1929] "Smart devices" are devices with advanced computing power and communication functions, and primarily refer to smartphones and tablets.

[1930] An "application" is a program or software that runs on a smart device.

[1931] An "alternate route" is a newly calculated route to avoid a no-entry area.

[1932] "Emotional state" refers to the user's emotional or psychological state, primarily stress level and satisfaction.

[1933] "Operation speed" refers to the speed at which the delivery robot moves.

[1934] A "route" is the path a delivery robot travels.

[1935] "Analysis" is the process of interpreting and making sense of data and information.

[1936] This invention relates to a system that enables autonomous mobile devices to autonomously determine no-entry areas and take evasive action. In addition, by combining this with an emotion engine that recognizes the user's emotions, it realizes more flexible and user-friendly operation management.

[1937] Server Operation

[1938] The server recognizes and maps no-entry areas using the following hardware and software:

[1939] 1. Receiving camera images

[1940] The server receives camera images periodically transmitted from the autonomous mobile device, and temporarily stores the received image data in a buffer.

[1941] 2. Prompt Generation

[1942] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[1943] Example prompt sentence:

[1944] Image: / path / to / image.jpg

[1945] Q: Is this area off-limits?

[1946] 3. Send to generative AI model

[1947] The server sends the generated prompt to the generative artificial intelligence model and waits for feedback, which includes information about the no-go area.

[1948] 4. Feedback Analysis

[1949] The feedback obtained from the generative AI model is analyzed to determine no-entry areas. The generative AI model is used for image analysis.

[1950] 5. Mapping No-Go Areas

[1951] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device, closely linking 2D image data with 3D point cloud data.

[1952] 6. Information Retention Period Management

[1953] No-entry area information is retained for a set period based on feedback from the generative AI model, after which it is deleted or updated from the database.

[1954] 7. Emotion Recognition with Emotion Engine

[1955] The emotion engine analyzes the user's speech and facial expressions and adjusts operation management based on that information.

[1956] Terminal (autonomous mobile device) operation

[1957] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action, and also adjusts its operation status based on feedback from the emotion engine.

[1958] 1. Camera image capture

[1959] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[1960] 2. Sending images

[1961] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[1962] 3. Receiving Instructions

[1963] Receive feedback from the server and obtain information about restricted areas.

[1964] 4. Taking evasive action

[1965] Based on the information about the no-entry areas, the autonomous mobile device recalculates an alternative route to the destination and avoids the no-entry areas.

[1966] 5. Receiving emotional feedback

[1967] It receives feedback from an emotion engine and adjusts its speed and route based on the user's stress level and emotional state.

[1968] User (delivery person or customer) behavior

[1969] The user manages and monitors the system.

[1970] 1. System Monitoring

[1971] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zones, in real time.

[1972] 2. Change settings

[1973] Users enter settings to prohibit entry to specific areas from the dashboard and send them to the server.

[1974] 3. Check operation

[1975] Users can check whether their autonomous mobile device is properly recognizing no-entry areas and taking evasive action, and adjust their route or operation schedule as necessary.

[1976] 4. Monitoring and setting emotion data

[1977] Users can monitor data from the emotion engine and adjust system settings based on their emotional state, for example slowing down the delivery robot if their stress levels are high.

[1978] As described above, the present invention provides means for making the operation management of autonomous mobile devices more efficient and safe.

[1979] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1980] Step 1:

[1981] The autonomous mobile device (terminal) captures camera images at regular intervals. The captured image data is temporarily stored in memory. The input is image data from the camera sensor, and the output is an image file in memory.

[1982] Step 2:

[1983] The device sends the image stored in memory to the server using an HTTP POST request. The input here is the image file in memory, and the output is the image data sent to the server. To ensure communication stability, a retry mechanism is included until the transmission is successful.

[1984] Step 3:

[1985] The server receives camera images sent from the autonomous mobile device and temporarily stores them in a buffer. The input is the image data from the device, and the output is the image data in the buffer.

[1986] Step 4:

[1987] The server generates a prompt to analyze the received image data. The prompt contains the image along with the text "Is this area off-limits?". The input is the image data in the buffer and a predefined question, and the output is the generated prompt.

[1988] Step 5:

[1989] The server sends the generated prompt to the AI ​​model and waits for feedback. The input is the prompt sentence, and the output is feedback about the no-entry area.

[1990] Step 6:

[1991] The server analyzes the feedback obtained from the generative AI model and determines the no-entry areas. The input is the feedback data returned from the generative AI model, and the output is the no-entry area determination result.

[1992] Step 7:

[1993] If a no-entry area is determined, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. The inputs are the no-entry area determination result, 2D image data, and 3D point cloud data, and the output is the updated map data.

[1994] Step 8:

[1995] The server manages information about restricted areas based on a retention period. The input is the information about restricted areas and the retention period setting, and the output is the data to be updated or deleted after the retention period has passed.

[1996] Step 9:

[1997] The server uses an emotion engine to analyze the user's speech and facial expressions, and adjusts operation management based on that information. The input is the user's speech data and facial expression data, and the output is analyzed emotional information. Based on this emotional information, the operation speed and route are adjusted.

[1998] Step 10:

[1999] The device receives feedback from the server and recalculates an alternative route based on the information on no-entry areas. The input is the no-entry area information from the server, and the output is the newly calculated alternative route.

[2000] Step 11:

[2001] The device continues moving to the destination according to the recalculated alternative route. The input is the new route information, and the output is the result of the movement.

[2002] Step 12:

[2003] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device in real time. The input is real-time data stored on the server, and the output is the operation status displayed on the user's screen.

[2004] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2005] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2006] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2007] [Fourth embodiment]

[2008] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2009] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2010] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2011] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2012] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2013] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2014] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2015] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2016] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2017] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2018] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2019] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2020] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2021] The present invention relates to a system for an autonomous mobile device to autonomously determine a no-entry area and take action to avoid it. The elements required to implement the present invention and their operations will be described below in detail.

[2022] Server Operation

[2023] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. The specific operation of the server is described below.

[2024] 1. Receiving camera images

[2025] The server receives camera images periodically transmitted from the autonomous mobile device, and the received image data is temporarily stored in a buffer.

[2026] 2. Prompt Generation

[2027] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[2028] 3. Send to generative AI model

[2029] The server sends the generated prompt to the artificial intelligence model and waits for a response, which includes information about the no-entry area.

[2030] 4. Feedback Analysis

[2031] The feedback from the generative artificial intelligence model is analyzed to determine whether the area is a no-entry area.

[2032] 5. Mapping No-Go Areas

[2033] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. A depth camera is used to link 3D point cloud data with 2D images to perform detailed mapping.

[2034] 6. Information Retention Period Management

[2035] Information about restricted areas is retained for a set period based on feedback from the generative AI model, and is then deleted or updated from the database after the period expires.

[2036] Terminal (autonomous mobile device) operation

[2037] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. The specific operation of the autonomous mobile device is described below.

[2038] 1. Camera image capture

[2039] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[2040] 2. Sending images

[2041] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[2042] 3. Receiving Instructions

[2043] Receive feedback from the server and obtain information about restricted areas.

[2044] 4. Taking evasive action

[2045] Based on the information on the restricted area, the autonomous mobile device recalculates an alternative route to the destination and avoids the restricted area. If it is unable to plot a route to the destination, it notifies the administrator.

[2046] User (facility administrator) actions

[2047] The user manages and monitors the system. The specific actions of the user are described below.

[2048] 1. System Monitoring

[2049] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[2050] 2. Change settings

[2051] Users can enter settings to prohibit entry to specific areas from the dashboard and send them to the server. Information on the designated area and period is sent to the server and reflected throughout the system.

[2052] 3. Check operation

[2053] The user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route and operation schedule.

[2054] Specific examples

[2055] The present invention will be described below with reference to specific examples.

[2056] Server example

[2057] 1. Receiving camera images

[2058] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[2059] 2. Prompt Generation

[2060] Generate a prompt containing the received image and format it in JSON format.

[2061] json

[2062] {

[2063] "image": " / tmp / captured_image.jpg",

[2064] "question": "Is this area off-limits?"

[2065] }

[2066] 3. Sending to the generative AI model and receiving feedback

[2067] This prompt is sent to the API endpoint of the generative AI model, and feedback is received saying, "This area is off-limits."

[2068] 4. Mapping No-Go Areas

[2069] The location of the no entry sign is converted into 3D coordinates and the map data of the autonomous mobile device is updated.

[2070] 5. Information Retention Period Management

[2071] The retention period information from the generative AI model, "This information is valid for one day," is registered in the database, and a deletion task is scheduled for one day later.

[2072] Examples of autonomous mobile devices

[2073] 1. Camera image capture

[2074] The autonomous mobile device uses a camera module to capture images within the facility.

[2075] 2. Sending images

[2076] Send the image data to the server using an HTTP POST request.

[2077] 3. Receiving Instructions

[2078] Receive feedback from the server that the area is off-limits and analyze the content.

[2079] 4. Taking evasive action

[2080] Based on information about restricted areas, a new route is calculated and travel to the destination is continued.

[2081] User Examples

[2082] 1. System Monitoring

[2083] Log in to the dashboard and monitor the operation status of the autonomous mobile device in real time.

[2084] 2. Change settings

[2085] Settings are made to temporarily prohibit entry to specific areas and sent to the server.

[2086] 3. Check operation

[2087] Check that the robot is avoiding restricted areas and adjust its route or schedule as necessary.

[2088] The processing flow will be explained below.

[2089] Server Processing Steps

[2090] Step 1:

[2091] The server waits for and receives camera images sent by the autonomous mobile device via HTTP requests, and temporarily stores the received image data in a buffer.

[2092] Step 2:

[2093] The server passes the received camera image to the prompt generation module, which generates a prompt containing the image and the text "Is this area off-limits?". The prompt is then formatted into JSON format to be sent to the generative AI model.

[2094] Step 3:

[2095] The server sends the formatted prompt via an HTTPS request to the API endpoint of the generative AI model and waits for feedback from the generative AI model.

[2096] Step 4:

[2097] The server receives feedback from the generative AI model in JSON format, including information such as "no entry areas," and analyzes the content to determine whether the area is a no entry area.

[2098] Step 5:

[2099] The server references spatial data linked to the camera's viewpoint to obtain coordinate information for the restricted area based on the feedback. It uses a depth camera or stereo method to link 3D data with 2D images and calculates detailed spatial coordinates.

[2100] Step 6:

[2101] The server adds the calculated spatial coordinates to the robot's map data and maps it as a new no-entry area.

[2102] Step 7:

[2103] The server stores the retention period information provided in the feedback of the generative AI model in a database, manages the retention period of the no-entry area information, and schedules a task to update the information when the retention period expires.

[2104] Processing steps of terminal (autonomous mobile device)

[2105] Step 1:

[2106] The autonomous mobile device captures images from the camera module at regular intervals and temporarily stores them in memory.

[2107] Step 2:

[2108] The autonomous mobile device sends the saved image to a specified server endpoint via an HTTP POST request. It also has a retry mechanism until the transmission is successful, ensuring communication stability.

[2109] Step 3:

[2110] The autonomous mobile device receives feedback from the server, analyzes the content, and obtains information about restricted areas.

[2111] Step 4:

[2112] The autonomous mobile device uses the received coordinate information of the no-entry area to autonomously calculate a new route and avoid the no-entry area.

[2113] Step 5:

[2114] The autonomous mobile device instructs the navigation module to find an avoidance route and continues moving to the destination. If the device cannot find a route to the destination, it notifies the administrator.

[2115] User (facility manager) processing steps

[2116] Step 1:

[2117] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[2118] Step 2:

[2119] Users can set specific areas to be off-limits as needed by entering the area's coordinates and the off-limits period into the form on the management screen and pressing the "Set" button to send the information to the server.

[2120] Step 3:

[2121] The user checks whether the autonomous mobile device is properly avoiding restricted areas and adjusts the robot's route and operation schedule as necessary.

[2122] Example 1

[2123] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2124] Autonomous mobile devices are required to accurately recognize no-entry areas and move around safely and efficiently. However, with conventional technology, the determination of no-entry areas and updating of map data are often done manually, making it difficult to respond in real time. In addition, managing the retention period for information about no-entry areas and taking appropriate evasive action are difficult, which can lead to reduced operational efficiency and safety for autonomous mobile devices.

[2125] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2126] In this invention, the server includes means for transmitting image data acquired by an imaging device mounted on the autonomous mobile device, means for generating the image data as a prompt, means for transmitting the prompt to a generative artificial intelligence model and receiving feedback regarding no-entry areas, means for analyzing the feedback and reflecting the no-entry areas on a map, means for managing information regarding the no-entry areas based on a retention period, and means for recalculating the route of the autonomous mobile device to avoid the no-entry areas. This enables the autonomous mobile device to autonomously recognize no-entry areas, update map data in real time, and take efficient avoidance actions.

[2127] An "autonomous mobile device" is a device equipped with sensors, cameras, and various control programs that moves and operates on its own judgment.

[2128] An "imaging device" is hardware for capturing images or video, such as a camera or depth sensor.

[2129] "Image data" is data that digitally represents visual information acquired by an imaging device.

[2130] A "prompt" is a data structure containing instructions or questions that are input to a generative AI model.

[2131] A "generative artificial intelligence model" is an AI system that uses machine learning and deep learning technologies to analyze and make predictions based on input data.

[2132] "Feedback" refers to the analysis results and response data returned as a response from the generative AI model.

[2133] A "no-entry area" is a specific spatial region to which access is restricted for the purpose of managing the safety of people and equipment and improving work efficiency.

[2134] "Reflecting on the map" means updating geographical data based on the acquired information and accurately showing the new information.

[2135] "Retention period" refers to the period for which information is considered valid, and means maintaining and managing the information until that period has elapsed.

[2136] "Recalculating a route" refers to recalculating the optimal route between the current location and the destination.

[2137] "3D point cloud data" is a collection of a series of points that make up a space, and each point is expressed in a 3D coordinate system (x, y, z).

[2138] The present invention relates to a system for an autonomous mobile device to autonomously determine a no-entry area and take action to avoid the area. Implementing the present invention requires the following specific steps.

[2139] Server Operation

[2140] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. Specific hardware used is a camera module (image capture device) and a depth sensor (a device for acquiring 3D point cloud data).

[2141] First, the server periodically receives camera images sent from the autonomous mobile device. This image data is temporarily stored in a buffer. The reception process is often performed using an HTTP POST request.

[2142] Next, the server generates a prompt to analyze the received image data. This prompt contains the path to the image file and the text "Is this area off-limits?". For example, the following prompt is generated:

[2143] Image: / tmp / captured_image.jpg

[2144] Q: Is this area off-limits?

[2145] The generated prompt is sent to the API endpoint of the generative AI model. After being sent to the API endpoint, the server waits for feedback from the generative AI model. The feedback includes information on whether the area is a no-entry area.

[2146] The server receives feedback from the AI ​​model and analyzes its contents. For example, it may receive feedback such as "This area is off-limits." If this feedback is analyzed and it is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. Specifically, it links the 3D point cloud data acquired from the depth camera with 2D images to perform detailed mapping.

[2147] Finally, the server manages the information about restricted areas based on a retention period. For example, if the generative AI model instructs the server to keep the information for one day, the server will register the information in the database and schedule a deletion task one day later.

[2148] Terminal (autonomous mobile device) operation

[2149] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. Specifically, it performs the following operations.

[2150] First, the autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in the device's memory.

[2151] Next, the autonomous mobile device transmits the stored image data to the server. If the transmission is unsuccessful, a retry mechanism is used to attempt retransmission.

[2152] Receives feedback from the server and obtains information on restricted areas. Analyzes the information, recalculates the route based on the restricted area information, and creates an alternative route to the destination. If no suitable route is found, notifies the administrator.

[2153] User (facility administrator) actions

[2154] Users use the management dashboard to monitor and manage the entire system. The specific actions of users are as follows:

[2155] First, the user accesses the management dashboard and monitors the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[2156] Next, the user enters settings for prohibiting entry to specific areas on the dashboard and sends them to the server. Information on the specified areas and period is sent to the server and reflected throughout the system.

[2157] Finally, the user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route or operation schedule.

[2158] Specific examples

[2159] For example, the following scenario is possible:

[2160] The server receives the image data from the "Autonomous Operation Sensor" and saves it as / tmp / captured_image.jpg. It generates a prompt containing the received image and formats it as follows:

[2161] Image: / tmp / captured_image.jpg

[2162] Q: Is this area off-limits?

[2163] This prompt is sent to the API endpoint of the generative AI model, and feedback is received stating, "This area is prohibited from entry." The location of the no-entry sign is converted into 3D coordinates, and the map data of the autonomous mobile device is updated. The generative AI model also registers the retention period information, "This information is valid for one day," in the database, and schedules a deletion task one day later.

[2164] The autonomous mobile device captures images using a camera module within the facility and sends them to the server via an HTTP POST request. It then receives feedback from the server, calculates a new route based on the information on "no entry areas," and continues moving toward its destination.

[2165] Users log in to the dashboard, monitor the operation status of the autonomous mobile device in real time, set up temporary prohibitions on certain areas, and send the information to the server. The system checks whether the robot is avoiding the prohibited areas and adjusts its route and schedule as necessary.

[2166] The above describes a specific embodiment of the invention. This system enables an autonomous mobile device to autonomously recognize no-entry areas, update map data in real time, and take efficient avoidance actions.

[2167] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2168] Server Operation

[2169] Step 1:

[2170] The server receives camera images from the autonomous mobile device.

[2171] Specific behavior:

[2172] The server receives the HTTP POST request and acquires the camera image data. The received image is temporarily stored in buffer memory.

[2173] Input: Camera image data sent from an autonomous mobile device.

[2174] Output: The image data will be saved in / tmp / captured_image.jpg.

[2175] Step 2:

[2176] The server generates a prompt to analyze the received image data.

[2177] Specific behavior:

[2178] The server generates a JSON prompt containing the file path of the received image and the text "Is this area off-limits?"

[2179] Input: The camera image saved as / tmp / captured_image.jpg and the question text.

[2180] Output: The generated prompt (e.g., {"image": " / tmp / captured_image.jpg", "question": "Is this area off-limits?"}).

[2181] Step 3:

[2182] The server sends the generated prompts to the generating artificial intelligence model.

[2183] Specific behavior:

[2184] The server sends an HTTP POST request to the API endpoint of the generated AI model.

[2185] Input: The generated prompt.

[2186] Output: The request sent to the API endpoint of the generative AI model.

[2187] Step 4:

[2188] The server receives and analyzes feedback from the generative AI model.

[2189] Specific behavior:

[2190] The server receives the HTTP response from the generated AI model and analyzes the feedback content, which is "No entry."

[2191] Input: Feedback from the generative AI model.

[2192] Output: No entry area determination result.

[2193] Step 5:

[2194] If the server determines that the area is a no-entry area, it calculates the spatial coordinates of the area and updates the map data.

[2195] Specific behavior:

[2196] The 3D point cloud data acquired by the depth camera is linked to the 2D images to calculate the precise location of no-entry areas and update the map data.

[2197] Input: No-entry area determination results, 3D point cloud data, and 2D image data.

[2198] Output: Updated map data.

[2199] Step 6:

[2200] The server manages the information about the no-entry areas based on a retention period.

[2201] Specific behavior:

[2202] Receive retention period information (e.g., "valid for 1 day") from the generated AI model and register it in the database. Schedule a deletion task after the specified period.

[2203] Input: No entry area information and retention period information.

[2204] Output: No entry area information registered in the database with retention period.

[2205] Terminal (autonomous mobile device) operation

[2206] Step 1:

[2207] The autonomous mobile device captures camera images at regular intervals.

[2208] Specific behavior:

[2209] The autonomous mobile device periodically activates the camera module to capture image data and temporarily store it in memory.

[2210] Input: A periodic timer event.

[2211] Output: Captured camera image data.

[2212] Step 2:

[2213] The autonomous mobile device transmits the captured images to a server.

[2214] Specific behavior:

[2215] The autonomous mobile device sends the stored image data to the server using an HTTP POST request. If the transmission is unsuccessful, the device attempts to retransmit the data using a retry mechanism.

[2216] Input: Captured camera image data.

[2217] Output: Image data sent to the server.

[2218] Step 3:

[2219] The autonomous mobile device receives feedback from the server.

[2220] Specific behavior:

[2221] Wait for feedback sent from the server and analyze it as soon as it is received.

[2222] Input: Feedback data from the server.

[2223] Output: Parsed feedback data.

[2224] Step 4:

[2225] The autonomous mobile device takes evasive action based on information about no-entry areas.

[2226] Specific behavior:

[2227] The system recalculates routes based on information about restricted areas and plans alternative routes to the destination. If no suitable route is found, it notifies the administrator.

[2228] Input: Parsed no-go area information.

[2229] Output: Recalculated route information or notification to administrator.

[2230] User (facility administrator) actions

[2231] Step 1:

[2232] Users access an administrative dashboard to monitor the status of their systems.

[2233] Specific behavior:

[2234] Through the dashboard, users can check the current location of the autonomous mobile device, its operating status, and the status of no-entry areas in real time.

[2235] Input: Dashboard access information and system status data.

[2236] Output: Visualized information showing the current state.

[2237] Step 2:

[2238] The user inputs settings to prohibit entry into specific areas.

[2239] Specific behavior:

[2240] The user designates a specific area as off-limits on the dashboard and sends that information to the server.

[2241] Input: User configuration input.

[2242] Output: No entry configuration information sent to the server.

[2243] Step 3:

[2244] The user checks the operating status of the autonomous mobile device.

[2245] Specific behavior:

[2246] The user monitors whether the autonomous mobile device recognizes no-entry areas and takes appropriate avoidance actions, adjusting the route or schedule as necessary.

[2247] Input: Dashboard traffic data.

[2248] Output: Check results and any necessary adjustments.

[2249] As described above, by providing detailed explanations for each processing step, including specific operations and inputs and outputs, we aim to ensure that the operation of this system is accurate and efficient.

[2250] (Application example 1)

[2251] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2252] Conventional autonomous mobile devices have limited capabilities for detecting and avoiding restricted areas, which has led to problems in ensuring safety, particularly in complex industrial facilities. There is also a risk that workers may mistakenly enter restricted or dangerous areas, which could lead to workplace accidents. Therefore, there is a need to improve safety by improving the ability of autonomous mobile devices to recognize restricted areas and by adding a real-time warning function to workers.

[2253] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2254] In this invention, the server includes means for periodically receiving camera images and temporarily storing them in a buffer, means for generating prompts for analyzing the received image data, means for sending the generated prompts to a generative artificial intelligence model and receiving feedback regarding no-entry areas, means for analyzing the feedback and mapping the no-entry areas, means for managing information regarding no-entry areas based on a retention period, and means for notifying smart devices worn by workers in industrial facilities in real time of information regarding no-entry areas and dangerous areas. This not only enables autonomous mobile devices to more accurately recognize and avoid no-entry areas, but also allows workers to be immediately warned of no-entry areas and dangerous areas, significantly improving safety.

[2255] An "autonomous mobile device" is a mechanical device that uses sensors to recognize its surrounding environment, automatically plans a route to its destination, and moves accordingly.

[2256] "Camera image" refers to visual information acquired by a camera mounted on an autonomous mobile device.

[2257] A "prompt" is a question generated based on camera images, and is text information provided to a generative artificial intelligence model to determine no-entry areas.

[2258] A "generative artificial intelligence model" is a system with machine learning algorithms used to generate answers to prompts.

[2259] "Feedback" is a response output from the generative artificial intelligence model, and includes information about no-entry areas.

[2260] "Mapping" is the process of converting the location of the no-entry area into spatial coordinates and recording them as map data for the autonomous mobile device.

[2261] "Retention period" refers to the period for which information regarding restricted areas is stored within the system.

[2262] "Industrial facility" refers to buildings and premises where manufacturing or other industrial activity is carried out.

[2263] "Smart equipment" refers to wearable devices worn by factory workers that have the ability to receive and display information about restricted and dangerous areas in real time.

[2264] "Real-time notification" means instantly transmitting information about restricted or dangerous areas to users.

[2265] A "hazardous area" is an area that poses a potential hazard to personnel or equipment.

[2266] The present invention relates to a system that enables autonomous mobile devices to autonomously determine restricted areas and take evasive action. In particular, the system can improve worker safety by linking with smart devices worn by workers in industrial facilities and notifying them of restricted or dangerous areas in real time.

[2267] Server Operation

[2268] The server receives camera images sent from the autonomous mobile device, analyzes them, and determines which areas are off-limits. The server uses the following hardware and software:

[2269] Hardware: high performance servers, storage devices, network interfaces

[2270] Software: image analysis algorithms, generative AI models, databases

[2271] The server first periodically receives camera images sent from the autonomous mobile device and stores them in a buffer. Next, it generates a prompt to analyze the received image data. This prompt includes the image data and the question, "Is this area off-limits?" The generated prompt is sent to the generative AI model, which receives feedback regarding the off-limits area. The received feedback is analyzed, and if it is determined to be an off-limits area, the location is converted into spatial coordinates and mapped. Finally, it sets a retention period for information regarding off-limits areas and manages them in a database.

[2272] Terminal (autonomous mobile device) operation

[2273] The autonomous mobile device works in cooperation with the server to recognize no-entry areas and take action to avoid them. The hardware and software of the terminal are as follows.

[2274] Hardware: Camera module, communication module, control unit

[2275] Software: Image capture software, communication protocols, path planning algorithms

[2276] The autonomous mobile device first captures images at regular intervals using a camera module. The captured image data is stored in memory and sent to a server. After receiving feedback from the server and obtaining information about no-entry areas, the device calculates an alternative route to the destination and continues moving while avoiding the no-entry areas.

[2277] User (factory workers and managers) actions

[2278] Factory workers and managers manage and monitor the system. The hardware and software used are as follows:

[2279] Hardware: Smart helmet, dashboard display

[2280] Software: Real-time monitoring application, notification system, configuration management interface

[2281] Factory workers wear smart helmets and receive real-time notifications of restricted or dangerous areas. For example, a message such as "This area is prohibited. Please avoid it" appears on the helmet's display. Managers can also use a dashboard to monitor the operation of the autonomous mobile devices and adjust zone settings and schedules.

[2282] Specific examples

[2283] Specific examples are shown below.

[2284] The server periodically receives and analyzes "image data within the facility."

[2285] An example of a prompt statement is: "Is this area off-limits?"

[2286] Workers can take immediate action when the message "Danger area. Please evacuate" appears on the display inside the smart helmet.

[2287] This provides a system that not only enables autonomous mobile devices to accurately recognize and avoid no-entry areas, but also improves the safety of factory workers.

[2288] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2289] Step 1:

[2290] The server periodically receives image data from the autonomous mobile device. The input is camera image data, which is temporarily stored in the server's buffer. This buffering allows for smooth subsequent data analysis.

[2291] Step 2:

[2292] The server generates a prompt based on the received image data. The input is the image data saved in the previous step, and the output is the prompt text. The server generates a prompt in JSON format that includes the image data and the text "Is this area off-limits?"

[2293] Step 3:

[2294] The server sends the generated prompt to the generative AI model and receives feedback. The input is the generated prompt, and the output is feedback from the generative AI model. The server determines whether the area is off-limits based on this feedback.

[2295] Step 4:

[2296] The server analyzes the feedback and maps the no-entry areas to spatial coordinates. The input is the feedback data from the generative AI model, and the output is the spatial coordinate data of the no-entry areas. The server links the 2D image data with the 3D point cloud data to obtain detailed location information.

[2297] Step 5:

[2298] The server manages information about restricted areas based on retention periods. The input is the spatial coordinate data of the restricted areas and retention period information, and the output is the updated database state. The server sets a schedule for deleting or updating the data after the set period.

[2299] Step 6:

[2300] The autonomous mobile device captures camera images at regular intervals. The input is real-time video data, and the output is captured still images. The images are stored in memory via the camera module.

[2301] Step 7:

[2302] The autonomous mobile device sends the captured image to the server. The input is the captured image data, and the output is an HTTP request to the server. The image data is sent to the server using the communication module.

[2303] Step 8:

[2304] The autonomous mobile device receives feedback from the server and acquires the no-entry area information. The input is the feedback data sent from the server, and the output is the analyzed no-entry area information. The information is acquired through the communication module.

[2305] Step 9:

[2306] The autonomous mobile device calculates a new route based on the no-entry area information and continues moving toward the destination. The input is the no-entry area information and the output is a new navigation route. A path planning algorithm is used to calculate the avoidance route.

[2307] Step 10:

[2308] Users wear smart devices and receive real-time notifications of restricted or dangerous areas. The input is a warning message from the server, and the output is the warning information displayed on the smart device's display. Users can immediately check this information and take evasive action.

[2309] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2310] The present invention relates to a system that enables an autonomous mobile device to autonomously determine no-entry areas and take evasive action. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it achieves more flexible and user-friendly operation management. Below, specific embodiments of each element and its operation for implementing the present invention will be described.

[2311] Server Operation

[2312] The server receives camera images from the autonomous mobile device, analyzes them, and maps information about restricted areas. It also acquires user emotional information and adjusts the settings of the entire system. The specific operation of the server is described below.

[2313] 1. Receiving camera images

[2314] The server receives camera images periodically transmitted from the autonomous mobile device, and the received image data is temporarily stored in a buffer.

[2315] 2. Prompt Generation

[2316] The server generates a prompt to analyze the received image data, which includes the image along with the text "Is this area off-limits?"

[2317] 3. Send to generative AI model

[2318] The server sends the generated prompt to the generative AI model and waits for feedback, which includes information about the no-go area.

[2319] 4. Feedback Analysis

[2320] The feedback from the generative artificial intelligence model is analyzed to determine whether the area is off-limits.

[2321] 5. Mapping No-Go Areas

[2322] If the area is determined to be a no-entry area, the server calculates the spatial coordinates of that area and updates the map data of the autonomous mobile device. A depth camera is used to link 3D point cloud data with 2D images to perform detailed mapping.

[2323] 6. Information Retention Period Management

[2324] Information about restricted areas is retained for a set period based on feedback from the generative AI model, and is then deleted or updated from the database after the period expires.

[2325] 7. Emotion Recognition with Emotion Engine

[2326] To obtain the user's emotional information, the emotion engine analyzes the user's speech and facial expressions, and adjusts the no-entry zone settings accordingly.

[2327] Terminal (autonomous mobile device) operation

[2328] The autonomous mobile device works in conjunction with the server to recognize no-entry areas and take evasive action. It also adjusts its operation status based on feedback from the emotion engine. The specific operation of the autonomous mobile device is described below.

[2329] 1. Camera image capture

[2330] The autonomous mobile device captures camera images at regular intervals, and the captured image data is temporarily stored in memory.

[2331] 2. Sending images

[2332] The autonomous mobile device transmits the stored images to the server, and is equipped with a retry mechanism until the transmission is successful, ensuring communication stability.

[2333] 3. Receiving Instructions

[2334] Receive feedback from the server and obtain information about restricted areas.

[2335] 4. Taking evasive action

[2336] Based on the information about the no-entry areas, the autonomous mobile device recalculates an alternative route to the destination and avoids the no-entry areas.

[2337] 5. Receiving emotional feedback

[2338] It receives feedback from an emotion engine and adjusts its speed and route based on the user's stress level and emotional state.

[2339] User (facility administrator) actions

[2340] The user manages and monitors the system. The specific actions of the user are described below.

[2341] 1. System Monitoring

[2342] Users can access the management dashboard to monitor the current location and operation status of the autonomous mobile device, as well as the status of no-entry zone settings, in real time.

[2343] 2. Change settings

[2344] Users can enter settings to prohibit entry to specific areas from the dashboard and send them to the server. Information on the designated area and period is sent to the server and reflected throughout the system.

[2345] 3. Check operation

[2346] The user can check whether the autonomous mobile device properly recognizes no-entry areas and takes evasive action. If necessary, the user can adjust the robot's route and operation schedule.

[2347] 4. Monitoring and setting emotion data

[2348] Users can monitor data from the emotion engine and adjust system settings based on their emotional state, for example slowing down the robot's speed if stress levels are high.

[2349] Specific examples

[2350] The present invention will be described below with reference to specific examples.

[2351] Server example

[2352] 1. Receiving camera images

[2353] The server receives image data from the "automatic driving sensor" and saves it as / tmp / captured_image.jpg.

[2354] 2. Generate and send prompts

[2355] Generate a prompt containing the received image and format it in JSON format.

[2356] json

[2357] {

[2358] "image": " / tmp / captured_image.jpg",

[2359] "question": "Is this area off-limits?"

[2360] }

[2361] This prompt is sent to the API endpoint of the generative AI model, and feedback is received saying, "This area is off-limits."

[2362] 3. Mapping No-Go Areas

[2363] The location of the no entry sign is converted into 3D coordinates and the map data of the autonomous mobile device is updated.

[2364] 4. Information Retention Period Management

[2365] The retention period information from the generative AI model, "This information is valid...

Claims

1. a means for transmitting image data acquired by a camera mounted on the autonomous mobile device; means for generating the image data as a prompt; means for sending the prompts to a generative artificial intelligence model and receiving feedback regarding the no-go areas; means for analyzing the feedback and mapping no-entry areas; a means for managing information about no-entry areas based on a retention period; A system including:

2. The system according to claim 1, further comprising means for linking 2D image data and 3D point cloud data to calculate spatial coordinates in mapping the no-entry area.

3. 2. The system according to claim 1, further comprising means for causing the autonomous mobile device to select an alternative route to the destination based on the information about the no-entry area.

Images