System

Abstract

To provide a system.SOLUTION: A system includes: means for mounting a display device carried by an operator; means for acquiring a video through a camera and a voice of the operator through a microphone; means for analyzing the acquired video to identify a type and position of a component; means for converting the acquired voice into text data; means for inputting an analysis result of an image and a recognition result of the voice into a generative AI model; means for the generative AI model into which the analysis result of the image and the recognition result of the voice are input to determine a component to be attached next and its procedure; means for outputting an instruction determined by the generative AI model as the text data; means for displaying an image indicating the position of the component on a display of the display device; and means for generating an instruction by voice and transmitting the instruction to the operator through a speaker.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional assembly and repair work on machines, vehicles, etc., workers had to decide for themselves what the next work procedure should be, which created challenges in improving work efficiency and preventing errors. [Means for solving the problem]

[0005] The present invention inputs images captured by a camera held by the worker and the worker's voice via a microphone into a generation AI, which then outputs instructions for the next work procedure to the worker in the form of voice or images, thereby improving work efficiency and preventing errors. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17]FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, a worker wears a head-mounted display and inputs video and audio to the generation AI through the display's built-in camera and microphone. The generation AI analyzes the video of the parts the worker should handle and the worker's audio (e.g., "Which part should I install next?") and determines the next work procedure. The instructions are conveyed to the worker through the display and speaker of the head-mounted display.

[0029] "Example 2"

[0030] As a specific example, consider the assembly of an automobile engine. A worker wears a head-mounted display and inputs images of engine parts and the voice saying, "Which part should I install next?" into the generative AI. The generative AI analyzes the video and voice and determines the next part to be installed and the order in which to install it. The instructions are communicated to the worker as an image showing the position of the parts on the head-mounted display screen and as a voice message over the speaker saying, "Install the piston next."

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: The worker wears a head-mounted display, which has a built-in camera and microphone.

[0034] Step 2: The worker starts the task, takes a picture of the part with the camera in the head-mounted display, and records a question about the task with the microphone. These images and voices are input to the generative AI.

[0035] Step 3: The generative AI analyzes the video and audio and determines the next steps to take.

[0036] Step 4: The generating AI communicates instructions for the next work procedure to the worker via the head-mounted display and speakers.

[0037] "Example 2"

[0038] Step 1: A worker starts assembling a car engine, takes pictures of the engine parts with a camera on a head-mounted display, and uses a microphone to record a question such as, "Which part should I install next?" These images and voices are input to the generative AI.

[0039] Step 2: The generative AI analyzes the video and audio to determine the next part to be installed and the order in which it should be installed.

[0040] Step 3: The generating AI communicates to the worker the position of the parts in the form of an image on the head-mounted display and audibly through the speaker, "Next, install the piston."

[0041] Example 1

[0042] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0043] Conventional work assistance systems have had difficulty efficiently understanding the next work procedure that a worker should perform. In addition, they were unable to analyze data collected through the camera and microphone carried by the worker in real time and provide appropriate instructions. This resulted in issues such as reduced work efficiency and an increased risk of mistakes.

[0044] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0045] In this invention, the server includes a means for inputting video and audio through a display device worn by a worker to the generative AI model, a means for the generative AI model to output instructions for the next work procedure to the worker by audio or image, a means for the display device to transmit collected data to the server, a means for the server to transmit the results of the generative AI model to the display device, and a means for the display device to present the work procedure to the worker. This allows the worker to grasp the next work procedure in real time, improving work efficiency and reducing the occurrence of errors.

[0046] A "worker" is a person who wears a display device and uses the system to perform a specific task.

[0047] A "display device" is a device such as a head-mounted display worn by a worker, which has the function of collecting video and audio and displaying instructions.

[0048] A "generative AI model" is an artificial intelligence model that analyzes collected video and audio data and determines the next work steps.

[0049] The "server" is a computer system whose role is to receive data sent from the display device, input it into the generative AI model, and send the results to the display device.

[0050] "Image" refers to visual information of the work environment and parts captured through the display device's camera.

[0051] "Voice" refers to acoustic information of the worker's comments and questions collected through the microphone of the display device.

[0052] "Input means" refers to a method or device for inputting the video and audio collected by the display device into the generative AI model.

[0053] "Means for output" refers to a method or device for providing the work procedures determined by the generative AI model to workers in the form of audio or images.

[0054] "Transmitting means" refers to a method or apparatus for the display device to transmit collected data to the server and for the server to transmit the results of the generative AI model to the display device.

[0055] "Presenting means" means a method or device by which a display device presents the results of the generative AI model to a human operator visually or audibly.

[0056] This invention is a system that collects video and audio through a display device worn by a worker and inputs them into a generative AI model to provide the next work procedure in real time. A specific embodiment of this system will be described below.

[0057] First, the user wears a head-mounted display (HMD). This HMD has a built-in camera and microphone, and has the function of collecting video and audio of the work environment. The HMD can be a commercially available device such as Microsoft (registered trademark) HoloLens (registered trademark) or Oculus Rift.

[0058] The device (HMD) uses a camera to capture images of the parts and work environment within the user's field of vision, and records the user's comments and questions with a microphone. This data is sent to a server in real time. Data is sent using wireless communication technologies such as Wi-Fi and Bluetooth.

[0059] The server inputs the received video and audio data into a generative AI model. The generative AI model is built using frameworks such as TENSORFLOW (registered trademark) and PyTorch. This generative AI model analyzes the video data to identify the part the user is handling, and analyzes the audio data to understand the user's question.

[0060] Once the generative AI model has determined the next work procedure, the result is sent from the server to the device. The device then presents the received work procedure to the user. Specifically, the HMD display shows an image and explanation of the next part to be installed, and the speaker issues audio instructions.

[0061] As a concrete example, consider the case where a user is assembling a car engine. The user is wearing an HMD and handling engine parts. When the user asks, "Which part should I install next?", the HMD's camera captures the current work situation and the microphone records the question. This data is sent to the server, where it is analyzed by the generative AI model. The generative AI model determines that the next part to be installed is a "piston" and returns this information to the server. The server sends the result to the terminal, which displays "Install the piston next" on the HMD's display and also plays the same instruction over the speaker.

[0062] Example prompt:

[0063] A worker is assembling an engine. Capture the current work status with a camera, analyze the worker's question "Which part should I install next?", and instruct the worker on which part to install next.

[0064] In this way, the system allows the server, terminals, and users to work together to efficiently carry out tasks.

[0065] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0066] Step 1:

[0067] The user puts on the HMD and begins working.

[0068] Specifically, the user puts on the HMD and turns it on. This activates the HMD's camera and microphone, allowing video and audio to be collected. The input is the video and audio of the parts and work environment within the user's field of vision, and the output is the data collected through the HMD's camera and microphone.

[0069] Step 2:

[0070] The terminal (HMD) collects video and audio.

[0071] Specifically, the HMD camera captures images of the parts and working environment within the user's field of vision, and the microphone records the user's comments and questions. The input is the images and audio of the parts and working environment within the user's field of vision, and the output is the collected video and audio data.

[0072] Step 3:

[0073] The terminal transmits the collected data to the server.

[0074] Specifically, the HMD compresses the collected video and audio data and transmits it to the server using wireless communication technologies such as Wi-Fi and Bluetooth. The input is the collected video and audio data, and the output is the data transmitted to the server.

[0075] Step 4:

[0076] The server inputs the data into a generative AI model.

[0077] Specifically, the server inputs the received video and audio data into a generative AI model. The generative AI model is built using frameworks such as TensorFlow and PyTorch. The input is the video and audio data sent to the server, and the output is the data input into the generative AI model.

[0078] Step 5:

[0079] A generative AI model analyzes the data and determines the next steps to take.

[0080] Specifically, the generative AI model analyzes video data to identify the part the user is handling, and analyzes audio data to understand the user's question. For example, if a user asks, "Which part should I install next?", the generative AI model analyzes the question and determines which part should be installed next. The input is the video and audio data input to the generative AI model, and the output is instructions for the next work procedure.

[0081] Step 6:

[0082] The server sends the results of the generated AI model to the device.

[0083] Specifically, the server sends the next work procedure determined by the generative AI model to the terminal. The data is compressed again and sent to the terminal via the network. The input is the result of the generative AI model, and the output is the work procedure instructions sent to the terminal.

[0084] Step 7:

[0085] The terminal presents the work procedure to the user.

[0086] Specifically, the terminal presents the received work procedure to the user. An image and explanation of the next part to be installed is shown on the HMD display, and instructions are given by voice from the speaker. For example, instructions such as "Install the piston next" are displayed. The input is the work procedure instructions sent to the terminal, and the output is the work procedure presented to the user.

[0087] (Application example 1)

[0088] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] In conventional assembly and repair work in factories, workers have to refer to the manual to confirm the next step, which reduces work efficiency. There is also a risk that the quality of work will decline if the worker makes a mistake in the next step. To solve these problems, a system that can provide instructions on work procedures in real time is needed.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0091] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, a means for the generation AI to analyze the voice command and image of the worker and determine the next work procedure, and a means for communicating the determined work procedure to the worker through the display and speaker of the head mounted display. This allows the worker to check the next work procedure in real time, making it possible to improve work efficiency and maintain work quality.

[0092] "Worker" refers to a person who assembles or repairs machines or parts in a factory or work site.

[0093] "Camera" refers to a device used to capture video and input the data into generative AI.

[0094] A "microphone" refers to a device for recording audio and inputting the data into the generative AI.

[0095] "Generative AI" refers to artificial intelligence that analyzes input video and audio and determines the next steps in the process.

[0096] A "head-mounted display" refers to a device worn by a worker to display and play back images and audio.

[0097] "Display" refers to a screen for visually displaying the work procedures determined by the generation AI.

[0098] "Speaker" refers to a device for playing back the work procedures determined by the generation AI in audio form.

[0099] "Voice command" refers to a voice instruction given by a worker to inquire about the next work procedure.

[0100] "Video" refers to visual information captured through a camera.

[0101] "Work procedure" refers to the specific operation or process that a worker should perform next.

[0102] The system for implementing this invention uses a camera and microphone held by a worker to input video and audio to a generation AI, which then determines the next work procedure and issues instructions to the worker. This system uses the following hardware and software.

[0103] Hardware

[0104] 1. Camera: The camera carried by the worker is used to capture footage of the work site and input the data into the generative AI.

[0105] 2. Microphone: The microphone held by the worker is used to record the worker's voice and input the data into the generative AI.

[0106] 3. Head-Mounted Display (HMD): The HMD worn by the worker is equipped with a display that visually displays the work procedures determined by the generation AI, and speakers that play them audibly.

[0107] software

[0108] 1. OpenCV: Used to acquire and process camera images.

[0109] 2. SpeechRecognition: Used to convert audio recorded from a microphone into text.

[0110] 3. Transformers: Uses generative AI models (e.g., GPT-3 (registered trademark)) to analyze input video and audio and determine the next steps.

[0111] System Operation

[0112] 1. Acquiring video and audio: The video of the work site and the voice of the worker are acquired through the camera and microphone held by the worker. The camera acquires video data using OpenCV, and the microphone converts the voice data into text using SpeechRecognition.

[0113] 2. Input to the generative AI: The captured video and audio data are input into the generative AI model, which uses Transformers to analyze the video and audio commands and determine the next steps.

[0114] 3. Output of work procedures: The work procedures determined by the generation AI are visually displayed on the HMD display and played audibly through the speaker.

[0115] Examples

[0116] For example, if a worker on a factory assembly line asks, "Which part should I install next?", a camera captures video of the work site and a microphone records the audio. The generative AI model uses the following prompt to determine the next step:

[0117] Example prompt:

[0118] Image: [Part image data], Voice command: "Which part should I install next?" What's the next step?

[0119] The generative AI model analyzes the prompt and determines specific work steps, such as "Next, install the screw." These instructions are displayed on the HMD display and communicated to the worker through the speaker.

[0120] In this way, workers can check the next work procedure in real time, improving work efficiency and maintaining work quality.

[0121] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0122] Step 1:

[0123] The server obtains images from the camera held by the worker. The camera captures images of the work site in real time and processes the data using OpenCV. The input is the image data from the camera, and the output is the processed image data.

[0124] Step 2:

[0125] The server captures the voice from the worker's microphone. The microphone records the worker's voice in real time and converts the data into text using SpeechRecognition. The input is the voice data from the microphone, and the output is the converted text data.

[0126] Step 3:

[0127] The server inputs the acquired video data and text data into the generative AI model. The generative AI model uses Transformers to analyze the video and voice commands and determine the next work steps. The input is video data and text data, and the output is text data regarding the next work steps.

[0128] Step 4:

[0129] The server displays the next work procedure obtained from the generative AI model on the display of a head-mounted display (HMD). The display visually shows the specific operation or process that the worker should perform next. The input is text data regarding the next work procedure, and the output is visual information displayed on the display.

[0130] Step 5:

[0131] The server plays the next work procedure obtained from the generative AI model aloud through the speaker of the head-mounted display (HMD). The speaker aloud tells the worker the specific operation or process that he or she should perform next. The input is text data regarding the next work procedure, and the output is audio information played from the speaker.

[0132] Step 6:

[0133] The user confirms the next steps through the HMD's display and speakers, and performs the work according to the instructions. The input is visual and audio information from the display and speakers, and the output is the work that the worker actually performs.

[0134] Example 2

[0135] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] With conventional work support systems, it was difficult to efficiently instruct workers on the next step in the work process. In addition, workers had to stop working to confirm the next step, which reduced work efficiency. Furthermore, it was difficult for workers to accurately grasp the location of parts, which meant there was a risk of installing the wrong parts.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0138] In this invention, the server includes a means for attaching a display device held by a worker, a means for acquiring an image through a camera and a voice of the worker through a microphone, a means for analyzing the acquired image with image analysis software and identifying the type and position of the part, a means for converting the acquired voice into text data with voice recognition software, a means for inputting the image analysis result and the voice recognition result into a generative AI model, a means for the generative AI model to determine the next part to be installed and the procedure, a means for outputting the instructions determined by the generative AI model as text data, a means for displaying an image showing the position of the part on the display of the display device, and a means for generating instructions by voice using voice synthesis software and communicating them to the worker through a speaker. This allows the worker to check the next work procedure without stopping his / her work and to accurately grasp the position of the part.

[0139] A "display device" is a device worn by a worker to provide visual information.

[0140] A "camera" is a device for taking images and acquiring the data.

[0141] A "microphone" is a device for recording sound and acquiring the data.

[0142] "Image analysis software" is software for analyzing acquired video data and extracting specific information.

[0143] "Voice recognition software" refers to software for converting acquired voice data into text data.

[0144] A "generative AI model" is an artificial intelligence model that analyzes input data and determines the next work procedure to be performed.

[0145] "Text data" is character information converted by voice recognition software.

[0146] A "display" is a screen built into a display device for displaying visual information.

[0147] "Speech synthesis software" means software for converting text data into speech and generating spoken instructions.

[0148] A "speaker" is a device for outputting speech generated by speech synthesis software.

[0149] This invention is a system in which a worker wears a display device, analyzes video and audio captured through a camera and microphone, and uses a generative AI model to instruct the next work procedure. A specific embodiment of this system will be described below.

[0150] Hardware and Software Used

[0151] Hardware: Display device (head-mounted display), camera, microphone, speaker

[0152] Software: Image analysis software, voice recognition software, generative AI models, voice synthesis software

[0153] System Operation

[0154] 1. The user wears a display device (head-mounted display) that has a built-in camera and microphone.

[0155] 2. The user takes a video of the engine parts with a camera and inputs into the microphone, "Which part should I install next?"

[0156] 3. The device captures the video captured by the camera and the audio recorded by the microphone.

[0157] 4. The video captured by the device is analyzed using image analysis software to identify the type and location of the engine parts.

[0158] 5. The voice captured by the device is converted into text data using voice recognition software.

[0159] 6. The device inputs the image analysis results and voice recognition results into the generative AI model.

[0160] 7. The generative AI model determines which part to install next and in what order.

[0161] 8. The instructions determined by the generative AI model are output as text data.

[0162] 9. The terminal receives instructions from the generative AI model and displays an image showing the location of the parts on the display device.

[0163] 10. The device uses speech synthesis software to generate a voice message saying, "Next, install the piston," and communicates this to the user through the speaker.

[0164] Examples

[0165] For example, say a user is assembling an engine. The user puts on the display device, takes pictures of the engine parts with the camera, and speaks into the microphone saying, "Which part should I install next?" The device captures the video and audio, uses image analysis software to identify the type and location of the part, and uses voice recognition software to convert the speech to text. This data is then input into a generative AI model that determines which part should be installed next and in what order. For example, an instruction might be generated: "Install the piston next." The device receives this instruction, displays an image showing the position of the piston on the display device's screen, and transmits the instruction via voice through the speaker.

[0166] Examples of prompt statements

[0167] "Analyze video and audio of engine parts to determine which part should be installed next and in what order. For example, if a piston needs to be installed next, generate an image showing its location and a voice command saying, 'Install the piston next.'"

[0168] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0169] Step 1:

[0170] The user wears a display device (head-mounted display).

[0171] Specifically, the user wears the display device on his / her head and the display is displayed within the field of vision. The display device has a built-in camera and microphone.

[0172] Step 2:

[0173] The user takes a picture of the engine parts with a camera and inputs into the microphone, "Which part should I install next?"

[0174] Specifically, the user points the camera at an engine part and takes a picture of it, while at the same time speaking into the microphone, "Which part should I install next?" The input data is video data and audio data.

[0175] Step 3:

[0176] The device captures video captured by the camera and audio recorded by the microphone.

[0177] In a specific operation, the terminal receives video data and audio data from the display device, the input data being the video data from the camera and the audio data from the microphone, and the output data being the captured video data and audio data.

[0178] Step 4:

[0179] The video captured by the device is analyzed using image analysis software to identify the type and location of engine parts.

[0180] Specifically, the device uses image analysis software to identify the type and location of engine parts from video data. The input data is the video data, and the output data is the type and location of the parts.

[0181] Step 5:

[0182] The voice captured by the device is converted into text data using voice recognition software.

[0183] Specifically, the terminal uses voice recognition software to convert voice data into text data, with the input data being voice data and the output data being text data.

[0184] Step 6:

[0185] The device inputs the image analysis results and voice recognition results into a generative AI model.

[0186] Specifically, the device inputs the image analysis results (type and position of parts) and voice recognition results (text data) into the generative AI model. The input data are the image analysis results and voice recognition results, and the output data is the input data for the generative AI model.

[0187] Step 7:

[0188] A generative AI model determines which parts to install next and in what order.

[0189] Specifically, the generative AI model analyzes the input data and determines the next part to be installed and the procedure for doing so. The input data is the results of image analysis and voice recognition, and the output data is the next part to be installed and the procedure for doing so.

[0190] Step 8:

[0191] The instructions determined by the generative AI model are output as text data.

[0192] Specifically, the instructions determined by the generative AI model are output to the terminal as text data. The input data is the next part to be installed and the procedure for installing it, and the output data is text data.

[0193] Step 9:

[0194] The terminal receives instructions from the generative AI model and displays an image showing the location of the parts on the display device's screen.

[0195] Specifically, the terminal receives instructions from the generative AI model and displays an image showing the position of the parts on the display of the display device. The input data is text data, and the output data is the image displayed on the display.

[0196] Step 10:

[0197] The device uses speech synthesis software to generate a voice message saying, "Next, install the piston," and communicates this to the user through the speaker.

[0198] In concrete terms, the terminal uses speech synthesis software to generate a voice message saying, "Next, install the piston," and transmits this message to the user through the speaker of the display device. The input data is text data, and the output data is the generated voice message.

[0199] (Application example 2)

[0200] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0201] In conventional assembly and repair work, workers had to spend a lot of time and effort trying to figure out which part to install next and the procedure for doing so. There was also a risk that workers would install the wrong part, which could have a negative impact on work efficiency and quality. This has led to a demand for improved work efficiency and quality.

[0202] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0203] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generating AI, a means for the generating AI to output instructions for the next work procedure to the worker as voice or image, a means for the generating AI to analyze the image and determine the next part to be installed and the procedure, a means for displaying an image showing the position of the part on the head mounted display, and a means for issuing voice instructions through a speaker. This enables the worker to accurately grasp the next part to be installed and the procedure in real time, thereby improving work efficiency and quality.

[0204] "Worker" means a person who assembles or repairs machines or vehicles in a factory or work site.

[0205] A "camera" is a device that captures images and records them as digital data.

[0206] A "microphone" is a device that records sound and saves it as digital data.

[0207] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[0208] "Input" refers to the input of data or information into a system.

[0209] "Output" refers to the system outputting the results of its processing.

[0210] A "means" is a method or device for achieving a specific purpose.

[0211] "Video" refers to visual information captured by a camera.

[0212] "Sound" refers to auditory information recorded by a microphone.

[0213] "Instructions" refers to commands to perform a particular action or task.

[0214] An "image" is digital data that represents visual information.

[0215] A "head-mounted display" is a display device that is worn on the head.

[0216] A "speaker" is a device for reproducing sound.

[0217] A "part" is an individual element that makes up a machine or vehicle.

[0218] A "procedure" is the order or method for performing a particular task.

[0219] As an embodiment of the present invention, we provide a system in which a factory robot is fitted with a head-mounted display and, when performing assembly work, uses generative AI to indicate in real time the next part to be installed and the procedure for doing so.

[0220] The server uses the following hardware and software:

[0221] Hardware: Camera, microphone, head-mounted display, speaker

[0222] Software: OpenCV (video capture and display), pyttsx3 (voice synthesis), transformers (generative AI model)

[0223] The server first uses a camera to capture the current assembly status and sends the video to the generation AI. The generation AI analyzes the video and audio and determines which parts should be installed next and in what order. It then displays an image showing the position of the parts on the head-mounted display and issues a voice command over the speaker, saying, "Install the piston next."

[0224] As a concrete example, when a robot is assembling an engine, a camera captures the current assembly status and sends a voice command such as "Which part should be installed next?" to the generation AI. The generation AI interprets this as "Install the piston next," displays an image showing the piston position on the head-mounted display, and issues a voice command over the speaker.

[0225] An example of a prompt is as follows:

[0226] "Please analyze the video showing the current assembly status and tell me which part should be installed next and in what order."

[0227] In this way, factory robots can efficiently carry out assembly work.

[0228] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0229] Step 1:

[0230] The server uses a camera to capture the current assembly status.

[0231] Input: Video data from the camera

[0232] Output: Captured video frames

[0233] How it works: The camera captures video in real time and acquires data frame by frame.

[0234] Step 2:

[0235] The server sends the captured video frames to the generation AI.

[0236] Input: Captured video frame

[0237] Output: Video data sent to the generation AI

[0238] What it does: Video frames are sent as digital data to a generative AI model.

[0239] Step 3:

[0240] The server uses a microphone to record the voice of the worker.

[0241] Input: Audio data from the microphone

[0242] Output: Recorded audio data

[0243] Specific operation: A microphone records the worker's voice in real time and stores it as digital data.

[0244] Step 4:

[0245] The server sends the recorded voice data to the generation AI.

[0246] Input: Recorded audio data

[0247] Output: Audio data sent to the generation AI

[0248] Specific operations: Audio data is sent as digital data to the generative AI model.

[0249] Step 5:

[0250] The generative AI analyzes video and audio data to determine the next part to be installed and the order in which it should be installed.

[0251] Input: Video data, audio data

[0252] Output: Text data of the next part to be installed and the procedure

[0253] Specific operation: The generative AI model analyzes video and audio and generates the appropriate parts and steps as text.

[0254] Step 6:

[0255] The server displays instructions from the generation AI on a head-mounted display.

[0256] Input: Text data of the next part to be installed and its procedure

[0257] Output: Image data displayed on the head-mounted display

[0258] Specific operation: Text data is converted into image data and displayed on a head-mounted display.

[0259] Step 7:

[0260] The server outputs instructions from the generation AI as audio through a speaker.

[0261] Input: Text data of the next part to be installed and its procedure

[0262] Output: Audio data output from the speaker

[0263] Specific operation: Convert text data into audio data and play it over a speaker.

[0264] In this way, the server instructs the worker on the next work procedure in real time, supporting efficient assembly work.

[0265] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0266] "Example 1"

[0267] In one embodiment of the present invention, a worker wears a head-mounted display. This display has a built-in camera and microphone, and an emotion engine. The worker starts a task, takes an image of the part with the camera of the head-mounted display, and records questions about the task with the microphone. These images and voices are input to the generation AI. The generation AI analyzes the images and voices and determines the next task procedure. Furthermore, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next task procedure based on the emotions. For example, if the emotion engine senses that the worker is stressed, it outputs an instruction to encourage the worker to take a break.

[0268] "Example 2"

[0269] In another embodiment of the present invention, a worker starts assembling an automobile engine, takes images of engine parts with a camera on a head-mounted display, and records a question with a microphone: "Which part should I install next?" These images and voices are input to a generative AI. The generative AI analyzes the images and voices and determines the next part to be installed and the order in which to install it. Furthermore, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on the emotions. For example, if the worker feels tired, the emotion engine outputs an instruction to encourage the worker to take a break.

[0270] The process flow of each embodiment will be described below.

[0271] "Example 1"

[0272] Step 1: The worker wears a head-mounted display, which has a built-in camera, microphone, and emotion engine.

[0273] Step 2: The worker begins the task, taking images of the part with the camera in the head-mounted display and recording questions about the task with the microphone.

[0274] Step 3: These images and audio are input into the generative AI.

[0275] Step 4: The generative AI analyzes the video and audio and determines the next steps.

[0276] Step 5: The emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on those emotions.

[0277] Step 6: For example, if the worker feels stressed, the emotion engine outputs an instruction to encourage the worker to take a break.

[0278] "Example 2"

[0279] Step 1: A worker begins assembling a car engine, taking images of the engine parts with a camera in a head-mounted display and recording a question with a microphone: "Which part should I install next?"

[0280] Step 2: These images and audio are input into the generative AI.

[0281] Step 3: The generative AI analyzes the video and audio to determine the next part to be installed and the order in which it should be installed.

[0282] Step 4: The emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on those emotions.

[0283] Step 5: For example, if a worker feels tired, the emotion engine outputs an instruction to encourage the worker to take a break.

[0284] Example 1

[0285] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0286] In conventional work support systems, it is sometimes difficult to give appropriate instructions on the next work procedure that a worker should perform. In addition, because the system gives uniform instructions without considering the worker's emotional state, it is difficult to reduce the worker's stress and fatigue. This reduces work efficiency and increases the possibility of work errors.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0288] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the voice of the worker and adjust the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate instructions for work procedures that take into account the emotional state of the worker, which is expected to improve work efficiency and reduce work errors.

[0289] "Worker" refers to a person who performs a particular task.

[0290] "Camera" refers to a device for capturing images.

[0291] "Microphone" refers to a device for recording sound.

[0292] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[0293] "Input" refers to entering data into a system.

[0294] "Output" refers to sending data out of the system.

[0295] An "emotion engine" is a system that recognizes emotions from voice and video, and generates appropriate instructions based on that information.

[0296] "Video" refers to visual information captured by a camera.

[0297] "Audio" refers to auditory information recorded by a microphone.

[0298] "Work procedure" refers to the specific steps or methods for performing a particular task.

[0299] "Instructions" refer to information intended to encourage a particular action.

[0300] A "system" refers to a series of devices or programs in which multiple elements work together.

[0301] This invention is a system in which a worker wears a head-mounted display and inputs video and audio to a generative AI via the display's built-in camera and microphone. The generative AI analyzes the video of the parts the worker should handle and the worker's audio (e.g., "Which part should I install next?") and determines the next work procedure. The instructions are communicated to the worker via the display and speaker of the head-mounted display.

[0302] Hardware and Software Used

[0303] Hardware: Head-mounted display (built-in camera, microphone, display, speaker)

[0304] Software: Generative AI models, emotion engines

[0305] Specific operation of the system

[0306] Subject: Server

[0307] The server receives video and audio data transmitted from the head-mounted display worn by the worker. This data is collected through the built-in camera and microphone. The server analyzes the received video data and identifies the part that the worker should handle. It also analyzes the audio data to understand the worker's question (e.g., "Which part should I install next?"). Based on the results of this analysis, the server determines the next work procedure and conveys those instructions to the worker via the display and speakers of the head-mounted display.

[0308] Subject: Terminal

[0309] The terminal (head-mounted display) is a device worn by the worker. The terminal uses a built-in camera and microphone to collect images of parts within the worker's field of vision and the worker's voice. The collected data is sent to the server in real time. When the terminal receives instructions from the server, it displays the instructions on the display and conveys them to the worker by voice through the speaker.

[0310] Subject: User

[0311] The user (worker) wears the head-mounted display while performing their work. As the user handles parts, the built-in camera captures the image, and the microphone records any questions about the work. When the user asks, "Which part should I install next?", the voice data is sent to the server. After receiving instructions from the server, the user checks the next work procedure through the display and speaker of the head-mounted display.

[0312] Examples

[0313] Example 1: Parts installation

[0314] When a user asks, "Which part should I attach next?", the microphone in the head-mounted display records the voice and sends it to the server. The server analyzes the video and audio data to identify the next part to be attached. For example, if the server instructs, "Attach the screw next," the instruction is displayed on the display of the head-mounted display and communicated by voice through the speaker. The user follows the instruction to attach the screw.

[0315] Example 2: Emotion-recognition-based break instruction

[0316] If a user says "I'm tired" while working, the microphone in the head-mounted display records the voice and sends it to the server. The emotion engine recognizes the user's stress from the voice. The emotion engine generates an instruction to encourage the user to take a break, and the server sends the instruction to the head-mounted display. The device displays the instruction "Let's take a short break" on the display and conveys it by voice through the speaker. The user follows the instruction and takes a break.

[0317] Examples of prompt statements

[0318] "If a worker wears a head-mounted display and asks what the next step is, please explain how the generative AI model would determine the next step and communicate it to the worker."

[0319] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0320] Step 1:

[0321] The user wears a head-mounted display.

[0322] Specifically, the user wears the head-mounted display and checks that the built-in camera and microphone are working properly, enabling the collection of video and audio.

[0323] Step 2:

[0324] The device collects video and audio.

[0325] Specifically, the device's built-in camera captures images of parts within the user's field of vision, and the built-in microphone records the user's voice. For example, if the user asks, "Which part should I install next?", that voice is recorded.

[0326] Input: Images of parts in the user's field of vision, user's voice

[0327] Output: Collected video and audio data

[0328] Step 3:

[0329] The terminal transmits the collected data to the server.

[0330] Specifically, the device transmits collected video and audio data in real time to the server, which then prepares the data for analysis.

[0331] Input: Collected video and audio data

[0332] Output: Data sent to the server

[0333] Step 4:

[0334] The server parses the data.

[0335] Specifically, the server analyzes the received video data to identify the part the user is handling. It also analyzes the voice data to understand the user's question. For example, if the user asks, "Which part should I install next?", the server analyzes and understands the question.

[0336] Input: Video and audio data sent to the server

[0337] Output: Analysis results (part identification, question understanding)

[0338] Step 5:

[0339] The server decides the next step to take.

[0340] Specifically, the server determines the next step based on the analysis results. For example, if the user determines that the part to be installed is a screw, the server generates the instruction "Install the screw next."

[0341] Input: Analysis results (part identification, understanding of question content)

[0342] Output: Next step instructions

[0343] Step 6:

[0344] The server sends the instruction to the terminal.

[0345] Specifically, the server sends instructions for the determined work procedure to the terminal, including the specific work that the user should do next.

[0346] Input: Next step instructions

[0347] Output: Instructions sent to the terminal

[0348] Step 7:

[0349] The terminal conveys the instructions to the user.

[0350] Specifically, the terminal conveys the instructions received from the server to the user. Specifically, the instructions are displayed on the head-mounted display and conveyed by voice through the speaker. For example, the instruction "Next, install the screws" is displayed on the display and conveyed by voice.

[0351] Input: Instructions sent to the terminal

[0352] Output: Instructions given to the user

[0353] (Application example 1)

[0354] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0355] In conventional factory work, workers had to refer to the manual or ask their superiors to confirm the next work procedure, which led to problems of reduced work efficiency. In addition, work instructions were given without considering the emotional state of the workers, which increased the workers' stress and increased the risk of mistakes and accidents.

[0356] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0357] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the voice of the worker and adjust the instructions for the next work procedure based on the emotions. This allows the worker to not only check the next work procedure in real time, but also to receive appropriate instructions according to his / her emotional state.

[0358] "Worker" refers to the person who actually performs the work in a factory or on-site.

[0359] "Camera" refers to a device that captures images and records them as digital data.

[0360] A "microphone" refers to a device that records sound and saves it as digital data.

[0361] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[0362] An "emotion engine" refers to a system that recognizes emotions from voice and video, and generates appropriate instructions based on the results.

[0363] "Video" refers to visual information captured through a camera.

[0364] "Speech" refers to auditory information captured through a microphone.

[0365] "Means" refers to a method or device for achieving a specific purpose.

[0366] A "system" refers to a set of devices or programs in which multiple elements function in conjunction with one another.

[0367] A "work procedure" refers to the series of steps or instructions required to complete a particular task.

[0368] "Output" refers to the output of results generated by the system to the outside.

[0369] "Input" refers to entering data into a system.

[0370] The system for implementing this invention captures video and audio using a camera and microphone held by the worker, and inputs this to the generation AI. The generation AI analyzes this data and determines the next work procedure. The emotion engine also recognizes emotions from the worker's voice and adjusts instructions based on those emotions.

[0371] Hardware and Software Used

[0372] Hardware: Head-mounted display (HMD), built-in camera, built-in microphone

[0373] Software: OpenCV (camera image acquisition), SpeechRecognition (voice acquisition), EmotionRecognizer (emotion recognition), AIDecisionMaker (generative AI model)

[0374] System Operation

[0375] The server acquires video through the HMD's camera and audio through the microphone. This data is input to the generative AI in real time. The generative AI analyzes the video and audio and determines the next work procedure. For example, if a worker asks, "Which part should I install next?" the generative AI analyzes the video and audio and issues the instruction, "Install the screws next."

[0376] Furthermore, the emotion engine recognizes the emotion of the worker from the voice. For example, if the worker is feeling stressed, the emotion engine will instruct the worker to "take a break." In this way, appropriate instructions are provided according to the worker's emotional state.

[0377] Examples

[0378] Take factory assembly work as an example. A worker wears an HMD, takes pictures of parts with the built-in camera, and records questions about the work with the built-in microphone. This data is input into the generative AI, which determines the next work procedure. For example, if a worker asks, "Which part should I attach next?" the generative AI will instruct, "Attach the screws next." If the worker is feeling stressed, the emotion engine will instruct, "Take a break."

[0379] Examples of prompt statements

[0380] An example of a prompt sentence to be input to the generative AI model is as follows.

[0381] The worker is asking, "Which part should I install next?" The video shows parts A and B. Please tell me the next step.

[0382] By inputting this prompt into a generative AI model, the next steps can be determined.

[0383] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0384] Step 1:

[0385] The server acquires images through the HMD camera. The image data from the camera is captured in real time and stored as digital data. This image data includes visual information of the parts handled by the worker and the work environment.

[0386] Step 2:

[0387] The server receives voice from the microphone of the HMD. The voice data from the microphone is captured in real time and stored as digital data. This voice data includes the worker's questions and instructions.

[0388] Step 3:

[0389] The server inputs the acquired video and audio data into the generation AI, which then analyzes the data and determines the next work procedure. Specifically, it identifies the type and location of parts from the video data, and understands the content of the worker's questions from the audio data.

[0390] Step 4:

[0391] The generative AI determines the next work procedure based on the analysis results and generates the instructions. For example, if a worker asks, "Which part should I attach next?", the generative AI generates the instruction, "Attach the screws next."

[0392] Step 5:

[0393] The server outputs instructions from the generation AI to the worker through the HMD's display and speakers. Visual instructions are displayed on the display, and audio instructions are played back from the speakers.

[0394] Step 6:

[0395] The server uses an emotion engine to recognize the worker's emotions from the voice data. The emotion engine analyzes the tone, speed, and content of the voice to determine whether the worker is feeling stressed.

[0396] Step 7:

[0397] The emotion engine recognizes the worker's emotions and adjusts the instructions of the generative AI based on the results. For example, if a worker is feeling stressed, the emotion engine will generate the instruction "Please take a break."

[0398] Step 8:

[0399] The server outputs the adjusted instructions to the worker again through the HMD display and speakers, allowing the worker to receive appropriate instructions according to their emotional state.

[0400] Example 2

[0401] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0402] Conventional work support systems have difficulty instructing workers on the next steps, and are particularly unable to provide instructions that take into account the worker's emotions and fatigue state. This leads to problems such as reduced work efficiency and increased worker fatigue and stress.

[0403] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0404] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for recognizing emotions from the voice of the worker using an emotion engine and adjusting the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate instructions for work procedures that take into account the emotions and fatigue state of the worker.

[0405] "Worker" refers to a person who performs work such as assembling or repairing machines, vehicles, etc.

[0406] "Camera" refers to a device for capturing images.

[0407] "Microphone" refers to a device for recording sound.

[0408] "Generative AI" refers to an artificial intelligence model that analyzes input data and generates an appropriate output.

[0409] An "emotion engine" refers to technology for recognizing emotions from data such as audio and video.

[0410] "Video" refers to visual information captured through a camera.

[0411] "Speech" refers to auditory information captured through a microphone.

[0412] "Input" refers to entering data into a system.

[0413] "Output" refers to sending data out of the system.

[0414] "Means" refers to a method or device for achieving a specific purpose.

[0415] "Work procedure" refers to the specific steps and methods for proceeding with work.

[0416] "Instructions" refers to communicating to workers the actions and procedures they should follow.

[0417] "Image" refers to a still image or video used to express visual information.

[0418] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0419] This invention is a system that uses a head-mounted display (HMD) to instruct workers on the next part to be installed and the procedure for installing it when assembling or repairing machines, vehicles, etc. The system is composed of a generative AI model that performs video and audio analysis, and an emotion engine that recognizes emotions.

[0420] Hardware and Software Used

[0421] Head-mounted display (HMD): Equipped with a camera and microphone, it captures the worker's visual and auditory information.

[0422] Server: Provides the computational resources to run the generative AI model and the emotion engine.

[0423] Generative AI model: Analyzes video and audio data to determine which part to install next and in what order.

[0424] Emotion engine: Recognizes emotions from a worker's voice and adjusts instructions for the next work step based on those emotions.

[0425] System Operation

[0426] 1. The user puts on the HMD:

[0427] The user puts on the HMD and starts assembling engine parts. The HMD has a built-in camera and microphone, which are used to record the progress of the work.

[0428] 2. The device acquires video and audio:

[0429] The device (HMD) uses a camera to capture images of engine parts in real time and a microphone to record the user's voice, for example when the user asks, "Which part should I install next?"

[0430] 3. The device sends the data to the server:

[0431] The device transmits captured video and recorded audio data to a server, where it is used for analysis by generative AI models and emotion engines.

[0432] 4. The server analyzes the data:

[0433] The server inputs the received video and audio data into a generative AI model. The generative AI model analyzes the state of the engine parts from the video and understands the user's question from the audio. For example, it recognizes from the video that the piston has not yet been installed, and understands from the audio that the user is asking about the next part to be installed.

[0434] 5. The server recognizes emotions:

[0435] The server uses an emotion engine to recognize emotions from the user's voice, for example if the user's voice sounds tired.

[0436] 6. The server generates the instruction:

[0437] The server determines the next part to be installed and the procedure based on the analysis results of the generative AI model and the recognition results of the emotion engine. For example, it generates an instruction such as "Install the piston next." If it recognizes that the user is tired, it also adds an instruction such as "Take a short break."

[0438] 7. The terminal will output the command:

[0439] The terminal receives instructions from the server, displays them on the HMD display as an image showing the position of the parts, and outputs voice instructions from the speaker such as "Next, attach the piston." If the user is tired, it also outputs voice instructions such as "Please take a short break."

[0440] Examples

[0441] Let's say a user is assembling an engine. The HMD camera captures video of the engine parts and asks, "Which part should I install next?" through the microphone. The device sends this data to the server. The server uses a generative AI model to analyze the video and audio, and generates the instruction, "Install the piston next." Furthermore, if the emotion engine recognizes fatigue from the user's voice, it also adds the instruction, "Please take a short break." The device communicates these instructions to the user through the HMD's display and speaker.

[0442] Examples of prompt statements

[0443] "We are currently assembling the engine. What is the next part to be installed and in what order? Please analyze the current video and audio data."

[0444] This system combines an HMD, a generative AI model, and an emotion engine to improve the user's work efficiency and provide flexible instructions according to the worker's condition.

[0445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0446] Step 1:

[0447] The user puts on the HMD and starts assembling engine parts. The user uses the camera and microphone of the HMD to record the progress of the work. Specifically, the user asks, "Which part should I install next?"

[0448] Input: User's audio and video

[0449] Output: Audio and video data recorded in the HMD

[0450] Step 2:

[0451] The device (HMD) captures images of engine parts and records audio. The camera captures images in real time, and the microphone records the user's questions. For example, the device captures images of engine parts and audio such as "Which part should I install next?"

[0452] Input: User's audio and video

[0453] Output: Shooting video data and recording audio data

[0454] Step 3:

[0455] The device transmits video and audio data to the server. The HMD transmits the captured data to the server, where it is used for analysis by the generative AI model and emotion engine.

[0456] Input: Recorded video data and recorded audio data

[0457] Output: Video and audio data sent to the server

[0458] Step 4:

[0459] The server analyzes the video and audio using a generative AI model. The server inputs the received video and audio data into the generative AI model, analyzes the state of the engine parts from the video, and understands the user's question from the audio. For example, it recognizes from the video that the piston has not yet been installed, and understands from the audio that the user is asking about the next part to be installed.

[0460] Input: Video and audio data sent to the server

[0461] Output: Analyzed part status and user question

[0462] Step 5:

[0463] The server uses the emotion engine to recognize the user's emotion. The server uses the emotion engine to recognize the emotion from the user's voice. For example, if the user's voice sounds tired, the server recognizes that emotion.

[0464] Input: User's voice data

[0465] Output: The perceived emotional state of the user.

[0466] Step 6:

[0467] The server generates instructions based on the analysis results and emotion recognition results. Based on the analysis results of the generative AI model and the recognition results of the emotion engine, the server determines the next part to be installed and the procedure for doing so. For example, it generates an instruction such as "Install the piston next." If it recognizes that the user is tired, it also adds an instruction such as "Take a short break."

[0468] Input: Analyzed part states and the user's emotional state

[0469] Output: Generated work instructions

[0470] Step 7:

[0471] The terminal displays and speaks instructions to the user. The terminal receives instructions from the server and displays them on the HMD display as an image showing the positions of parts. It also outputs voice instructions from the speaker, such as "Next, attach the piston." If the user is tired, it also outputs voice instructions such as "Please take a short break."

[0472] Input: Generated work order

[0473] Output: Images displayed on the HMD display and audio instructions output from the speaker.

[0474] (Application example 2)

[0475] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0476] Conventional assembly work support systems have difficulty instructing workers on the next work procedure, which reduces work efficiency. In addition, they are unable to give instructions that take into account the worker's fatigue and stress, which can have a negative impact on the worker's health and work quality. To solve these problems, a system that recognizes the worker's emotions and issues appropriate instructions based on them is needed.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0478] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the worker's voice and adjust the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate work instructions that take into account the emotions of the worker, improving work efficiency and maintaining the health of the worker.

[0479] A "camera" is a device for capturing images.

[0480] A "microphone" is a device for capturing sound.

[0481] "Generative AI" is artificial intelligence that analyzes input data and generates appropriate output.

[0482] An "emotion engine" is a system that recognizes emotions from data such as audio and video, and generates appropriate instructions based on those emotions.

[0483] A "task procedure" is the sequence of steps required to complete a particular task.

[0484] "Voice" refers to words and sounds made by the worker.

[0485] "Video" is visual information captured through a camera.

[0486] "Output" refers to the provision of instructions and information generated by the system to workers.

[0487] "Input" refers to data or information entered into a system.

[0488] A "system" is a configuration in which multiple devices and software work together to achieve a specific function.

[0489] The system for implementing this invention captures video and audio using a camera and microphone held by the worker, and inputs them into the generation AI. The generation AI analyzes the next work procedure and outputs it to the worker as audio or images. The emotion engine also recognizes emotions from the worker's voice and adjusts the instructions for the next work procedure based on those emotions.

[0490] Hardware and Software Used

[0491] Hardware:

[0492] Camera (e.g. Logitech C920)

[0493] Microphone (e.g. Blue Yeti)

[0494] Head-mounted displays (e.g. Microsoft HoloLens)

[0495] software:

[0496] OpenCV (for image capture)

[0497] SpeechRecognition (for voice capture)

[0498] Transformers (using generative AI and sentiment analysis models)

[0499] Data processing and calculation

[0500] The server receives the video captured through the camera and the audio captured through the microphone. These data are input to the generative AI model, which generates the next steps to be taken. The generated instructions are displayed on the head-mounted display or communicated to the worker by voice. In addition, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions based on those emotions.

[0501] Examples

[0502] When a worker asks, "Which part should I install next?", a camera takes a picture of the engine part and inputs it into the generative AI model. The generative AI model generates an instruction, such as "Install the piston next," and displays it on the head-mounted display. Furthermore, if the emotion engine senses fatigue from the worker's voice, it issues the instruction, "Please take a break."

[0503] Examples of prompt statements

[0504] Image: <Engine parts image>

[0505] Q: What part should I install next?

[0506] Please tell me what parts I should install next and what steps to take.

[0507] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0508] Step 1:

[0509] A user uses a camera to capture video of an engine part.

[0510] Input: Physical condition of engine parts

[0511] Output: Captured image data of engine parts

[0512] Specific operation: A camera (e.g. Logitech C920) is used to capture images of engine parts. The image data is sent to the server in real time.

[0513] Step 2:

[0514] The user uses a microphone to capture audio saying, "Which part should I install next?"

[0515] Input: User's voice

[0516] Output: Captured audio data

[0517] What it does: Use a microphone (e.g. Blue Yeti) to record the user's question. The audio data is sent to the server in real time.

[0518] Step 3:

[0519] The server inputs the captured video and audio into a generative AI model.

[0520] Input: Video data, audio data

[0521] Output: Input data for the generative AI model

[0522] How it works: The server receives the captured video and audio and formats the data for input into a generative AI model (e.g. GPT-3.5-turbo).

[0523] Step 4:

[0524] The generative AI model analyzes the next work step and generates instructions.

[0525] Input: Input data (video data, audio data)

[0526] Output: Work procedure instructions

[0527] Specific operation: The generative AI model analyzes the input data and determines which parts to install next and in what order, for example generating instructions such as "Install the piston next."

[0528] Step 5:

[0529] The server displays the generated instructions on a head-mounted display or conveys them to the user by voice.

[0530] Input: Work procedure instructions

[0531] Output: Display or audio instructions

[0532] Specific operation: The server displays the generated instructions on the display of a head-mounted display (e.g., Microsoft HoloLens) or conveys them to the user audibly through the speakers.

[0533] Step 6:

[0534] The server uses an emotion engine to recognize emotions from the user's voice.

[0535] Input: User's voice data

[0536] Output: Emotion recognition result

[0537] Specific operation: The emotion engine analyzes the user's voice data and recognizes the user's emotions (e.g., tiredness, stress).

[0538] Step 7:

[0539] The server adjusts the instructions for the next work step based on the emotion.

[0540] Input: Emotion recognition results

[0541] Output: Coordinated work instructions

[0542] Specific Actions: Based on the results of the emotion engine, the server adjusts the next steps of the work. For example, if the server detects that the user is tired, it will give the instruction "Please take a break."

[0543] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0544] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0545] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0546] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0547] [Second embodiment]

[0548] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0549] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0550] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0551] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0552] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0553] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0554] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0555] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0556] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0557] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0558] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0559] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0560] "Example 1"

[0561] In one embodiment of the present invention, a worker wears a head-mounted display and inputs video and audio to the generation AI through the display's built-in camera and microphone. The generation AI analyzes the video of the parts the worker should handle and the worker's audio (e.g., "Which part should I install next?") and determines the next work procedure. The instructions are conveyed to the worker through the display and speaker of the head-mounted display.

[0562] "Example 2"

[0563] As a specific example, consider the assembly of an automobile engine. A worker wears a head-mounted display and inputs images of engine parts and the voice saying, "Which part should I install next?" into the generative AI. The generative AI analyzes the video and voice and determines the next part to be installed and the order in which to install it. The instructions are communicated to the worker as an image showing the position of the parts on the head-mounted display screen and as a voice message over the speaker saying, "Install the piston next."

[0564] The process flow of each embodiment will be described below.

[0565] "Example 1"

[0566] Step 1: The worker wears a head-mounted display, which has a built-in camera and microphone.

[0567] Step 2: The worker starts the task, takes a picture of the part with the camera in the head-mounted display, and records a question about the task with the microphone. These images and voices are input to the generative AI.

[0568] Step 3: The generative AI analyzes the video and audio and determines the next steps to take.

[0569] Step 4: The generating AI communicates instructions for the next work procedure to the worker via the head-mounted display and speakers.

[0570] "Example 2"

[0571] Step 1: A worker starts assembling a car engine, takes pictures of the engine parts with a camera on a head-mounted display, and uses a microphone to record a question such as, "Which part should I install next?" These images and voices are input to the generative AI.

[0572] Step 2: The generative AI analyzes the video and audio to determine the next part to be installed and the order in which it should be installed.

[0573] Step 3: The generating AI communicates to the worker the position of the parts in the form of an image on the head-mounted display and audibly through the speaker, "Next, install the piston."

[0574] Example 1

[0575] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0576] Conventional work assistance systems have had difficulty efficiently understanding the next work procedure that a worker should perform. In addition, they were unable to analyze data collected through the camera and microphone carried by the worker in real time and provide appropriate instructions. This resulted in issues such as reduced work efficiency and an increased risk of mistakes.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0578] In this invention, the server includes a means for inputting video and audio through a display device worn by a worker to the generative AI model, a means for the generative AI model to output instructions for the next work procedure to the worker by audio or image, a means for the display device to transmit collected data to the server, a means for the server to transmit the results of the generative AI model to the display device, and a means for the display device to present the work procedure to the worker. This allows the worker to grasp the next work procedure in real time, improving work efficiency and reducing the occurrence of errors.

[0579] A "worker" is a person who wears a display device and uses the system to perform a specific task.

[0580] A "display device" is a device such as a head-mounted display worn by a worker, which has the function of collecting video and audio and displaying instructions.

[0581] A "generative AI model" is an artificial intelligence model that analyzes collected video and audio data and determines the next work steps.

[0582] The "server" is a computer system whose role is to receive data sent from the display device, input it into the generative AI model, and send the results to the display device.

[0583] "Image" refers to visual information of the work environment and parts captured through the display device's camera.

[0584] "Voice" refers to acoustic information of the worker's comments and questions collected through the microphone of the display device.

[0585] "Input means" refers to a method or device for inputting the video and audio collected by the display device into the generative AI model.

[0586] "Means for output" refers to a method or device for providing the work procedures determined by the generative AI model to workers in the form of audio or images.

[0587] "Transmitting means" refers to a method or apparatus for the display device to transmit collected data to the server and for the server to transmit the results of the generative AI model to the display device.

[0588] "Presenting means" means a method or device by which a display device presents the results of the generative AI model to a human operator visually or audibly.

[0589] This invention is a system that collects video and audio through a display device worn by a worker and inputs them into a generative AI model to provide the next work procedure in real time. A specific embodiment of this system will be described below.

[0590] First, the user wears a head-mounted display (HMD). This HMD has a built-in camera and microphone, and has the function of collecting video and audio of the work environment. The HMD can be a commercially available device such as Microsoft HoloLens or Oculus Rift.

[0591] The device (HMD) uses a camera to capture images of the parts and work environment within the user's field of vision, and records the user's comments and questions with a microphone. This data is sent to a server in real time. Data is sent using wireless communication technologies such as Wi-Fi and Bluetooth.

[0592] The server inputs the received video and audio data into a generative AI model, which is built using frameworks such as TensorFlow and PyTorch. The generative AI model analyzes the video data to identify the part the user is handling, and analyzes the audio data to understand the user's question.

[0593] Once the generative AI model has determined the next work procedure, the result is sent from the server to the device. The device then presents the received work procedure to the user. Specifically, the HMD display shows an image and explanation of the next part to be installed, and the speaker issues audio instructions.

[0594] As a concrete example, consider the case where a user is assembling a car engine. The user is wearing an HMD and handling engine parts. When the user asks, "Which part should I install next?", the HMD's camera captures the current work situation and the microphone records the question. This data is sent to the server, where it is analyzed by the generative AI model. The generative AI model determines that the next part to be installed is a "piston" and returns this information to the server. The server sends the result to the terminal, which displays "Install the piston next" on the HMD's display and also plays the same instruction over the speaker.

[0595] Example prompt:

[0596] A worker is assembling an engine. Capture the current work status with a camera, analyze the worker's question "Which part should I install next?", and instruct the worker on which part to install next.

[0597] In this way, the system allows the server, terminals, and users to work together to efficiently carry out tasks.

[0598] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0599] Step 1:

[0600] The user puts on the HMD and begins working.

[0601] Specifically, the user puts on the HMD and turns it on. This activates the HMD's camera and microphone, allowing video and audio to be collected. The input is the video and audio of the parts and work environment within the user's field of vision, and the output is the data collected through the HMD's camera and microphone.

[0602] Step 2:

[0603] The terminal (HMD) collects video and audio.

[0604] Specifically, the HMD camera captures images of the parts and working environment within the user's field of vision, and the microphone records the user's comments and questions. The input is the images and audio of the parts and working environment within the user's field of vision, and the output is the collected video and audio data.

[0605] Step 3:

[0606] The terminal transmits the collected data to the server.

[0607] Specifically, the HMD compresses the collected video and audio data and transmits it to the server using wireless communication technologies such as Wi-Fi and Bluetooth. The input is the collected video and audio data, and the output is the data transmitted to the server.

[0608] Step 4:

[0609] The server inputs the data into a generative AI model.

[0610] Specifically, the server inputs the received video and audio data into a generative AI model. The generative AI model is built using frameworks such as TensorFlow and PyTorch. The input is the video and audio data sent to the server, and the output is the data input into the generative AI model.

[0611] Step 5:

[0612] A generative AI model analyzes the data and determines the next steps to take.

[0613] Specifically, the generative AI model analyzes video data to identify the part the user is handling, and analyzes audio data to understand the user's question. For example, if a user asks, "Which part should I install next?", the generative AI model analyzes the question and determines which part should be installed next. The input is the video and audio data input to the generative AI model, and the output is instructions for the next work procedure.

[0614] Step 6:

[0615] The server sends the results of the generated AI model to the device.

[0616] Specifically, the server sends the next work procedure determined by the generative AI model to the terminal. The data is compressed again and sent to the terminal via the network. The input is the result of the generative AI model, and the output is the work procedure instructions sent to the terminal.

[0617] Step 7:

[0618] The terminal presents the work procedure to the user.

[0619] Specifically, the terminal presents the received work procedure to the user. An image and explanation of the next part to be installed is shown on the HMD display, and instructions are given by voice from the speaker. For example, instructions such as "Install the piston next" are displayed. The input is the work procedure instructions sent to the terminal, and the output is the work procedure presented to the user.

[0620] (Application example 1)

[0621] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0622] In conventional assembly and repair work in factories, workers have to refer to the manual to confirm the next step, which reduces work efficiency. There is also a risk that the quality of work will decline if the worker makes a mistake in the next step. To solve these problems, a system that can provide instructions on work procedures in real time is needed.

[0623] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0624] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, a means for the generation AI to analyze the voice command and image of the worker and determine the next work procedure, and a means for communicating the determined work procedure to the worker through the display and speaker of the head mounted display. This allows the worker to check the next work procedure in real time, making it possible to improve work efficiency and maintain work quality.

[0625] "Worker" refers to a person who assembles or repairs machines or parts in a factory or work site.

[0626] "Camera" refers to a device used to capture video and input the data into generative AI.

[0627] A "microphone" refers to a device for recording audio and inputting the data into the generative AI.

[0628] "Generative AI" refers to artificial intelligence that analyzes input video and audio and determines the next steps in the process.

[0629] A "head-mounted display" refers to a device worn by a worker to display and play back images and audio.

[0630] "Display" refers to a screen for visually displaying the work procedures determined by the generation AI.

[0631] "Speaker" refers to a device for playing back the work procedures determined by the generation AI in audio form.

[0632] "Voice command" refers to a voice instruction given by a worker to inquire about the next work procedure.

[0633] "Video" refers to visual information captured through a camera.

[0634] "Work procedure" refers to the specific operation or process that a worker should perform next.

[0635] The system for implementing this invention uses a camera and microphone held by a worker to input video and audio to a generation AI, which then determines the next work procedure and issues instructions to the worker. This system uses the following hardware and software.

[0636] Hardware

[0637] 1. Camera: The camera carried by the worker is used to capture footage of the work site and input the data into the generative AI.

[0638] 2. Microphone: The microphone held by the worker is used to record the worker's voice and input the data into the generative AI.

[0639] 3. Head-Mounted Display (HMD): The HMD worn by the worker is equipped with a display that visually displays the work procedures determined by the generation AI, and speakers that play them audibly.

[0640] software

[0641] 1. OpenCV: Used to acquire and process camera images.

[0642] 2. SpeechRecognition: Used to convert audio recorded from a microphone into text.

[0643] 3. Transformers: Uses generative AI models (e.g. GPT-3) to analyze input video and audio and determine the next steps.

[0644] System Operation

[0645] 1. Acquiring video and audio: The video of the work site and the voice of the worker are acquired through the camera and microphone held by the worker. The camera acquires video data using OpenCV, and the microphone converts the voice data into text using SpeechRecognition.

[0646] 2. Input to the generative AI: The captured video and audio data are input into the generative AI model, which uses Transformers to analyze the video and audio commands and determine the next steps.

[0647] 3. Output of work procedures: The work procedures determined by the generation AI are visually displayed on the HMD display and played audibly through the speaker.

[0648] Examples

[0649] For example, if a worker on a factory assembly line asks, "Which part should I install next?", a camera captures video of the work site and a microphone records the audio. The generative AI model uses the following prompt to determine the next step:

[0650] Example prompt:

[0651] Image: [Part image data], Voice command: "Which part should I install next?" What's the next step?

[0652] The generative AI model analyzes the prompt and determines specific work steps, such as "Next, install the screw." These instructions are displayed on the HMD display and communicated to the worker through the speaker.

[0653] In this way, workers can check the next work procedure in real time, improving work efficiency and maintaining work quality.

[0654] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0655] Step 1:

[0656] The server obtains images from the camera held by the worker. The camera captures images of the work site in real time and processes the data using OpenCV. The input is the image data from the camera, and the output is the processed image data.

[0657] Step 2:

[0658] The server captures the voice from the worker's microphone. The microphone records the worker's voice in real time and converts the data into text using SpeechRecognition. The input is the voice data from the microphone, and the output is the converted text data.

[0659] Step 3:

[0660] The server inputs the acquired video data and text data into the generative AI model. The generative AI model uses Transformers to analyze the video and voice commands and determine the next work steps. The input is video data and text data, and the output is text data regarding the next work steps.

[0661] Step 4:

[0662] The server displays the next work procedure obtained from the generative AI model on the display of a head-mounted display (HMD). The display visually shows the specific operation or process that the worker should perform next. The input is text data regarding the next work procedure, and the output is visual information displayed on the display.

[0663] Step 5:

[0664] The server plays the next work procedure obtained from the generative AI model aloud through the speaker of the head-mounted display (HMD). The speaker aloud tells the worker the specific operation or process that he or she should perform next. The input is text data regarding the next work procedure, and the output is audio information played from the speaker.

[0665] Step 6:

[0666] The user confirms the next steps through the HMD's display and speakers, and performs the work according to the instructions. The input is visual and audio information from the display and speakers, and the output is the work that the worker actually performs.

[0667] Example 2

[0668] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0669] With conventional work support systems, it was difficult to efficiently instruct workers on the next step in the work process. In addition, workers had to stop working to confirm the next step, which reduced work efficiency. Furthermore, it was difficult for workers to accurately grasp the location of parts, which meant there was a risk of installing the wrong parts.

[0670] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0671] In this invention, the server includes a means for attaching a display device held by a worker, a means for acquiring an image through a camera and a voice of the worker through a microphone, a means for analyzing the acquired image with image analysis software and identifying the type and position of the part, a means for converting the acquired voice into text data with voice recognition software, a means for inputting the image analysis result and the voice recognition result into a generative AI model, a means for the generative AI model to determine the next part to be installed and the procedure, a means for outputting the instructions determined by the generative AI model as text data, a means for displaying an image showing the position of the part on the display of the display device, and a means for generating instructions by voice using voice synthesis software and communicating them to the worker through a speaker. This allows the worker to check the next work procedure without stopping his / her work and to accurately grasp the position of the part.

[0672] A "display device" is a device worn by a worker to provide visual information.

[0673] A "camera" is a device for taking images and acquiring the data.

[0674] A "microphone" is a device for recording sound and acquiring the data.

[0675] "Image analysis software" is software for analyzing acquired video data and extracting specific information.

[0676] "Voice recognition software" refers to software for converting acquired voice data into text data.

[0677] A "generative AI model" is an artificial intelligence model that analyzes input data and determines the next work procedure to be performed.

[0678] "Text data" is character information converted by voice recognition software.

[0679] A "display" is a screen built into a display device for displaying visual information.

[0680] "Speech synthesis software" means software for converting text data into speech and generating spoken instructions.

[0681] A "speaker" is a device for outputting speech generated by speech synthesis software.

[0682] This invention is a system in which a worker wears a display device, analyzes video and audio captured through a camera and microphone, and uses a generative AI model to instruct the next work procedure. A specific embodiment of this system will be described below.

[0683] Hardware and Software Used

[0684] Hardware: Display device (head-mounted display), camera, microphone, speaker

[0685] Software: Image analysis software, voice recognition software, generative AI models, voice synthesis software

[0686] System Operation

[0687] 1. The user wears a display device (head-mounted display) that has a built-in camera and microphone.

[0688] 2. The user takes a video of the engine parts with a camera and inputs into the microphone, "Which part should I install next?"

[0689] 3. The device captures the video captured by the camera and the audio recorded by the microphone.

[0690] 4. The video captured by the device is analyzed using image analysis software to identify the type and location of the engine parts.

[0691] 5. The voice captured by the device is converted into text data using voice recognition software.

[0692] 6. The device inputs the image analysis results and voice recognition results into the generative AI model.

[0693] 7. The generative AI model determines which part to install next and in what order.

[0694] 8. The instructions determined by the generative AI model are output as text data.

[0695] 9. The terminal receives instructions from the generative AI model and displays an image showing the location of the parts on the display device.

[0696] 10. The device uses speech synthesis software to generate a voice message saying, "Next, install the piston," and communicates this to the user through the speaker.

[0697] Examples

[0698] For example, say a user is assembling an engine. The user puts on the display device, takes pictures of the engine parts with the camera, and speaks into the microphone saying, "Which part should I install next?" The device captures the video and audio, uses image analysis software to identify the type and location of the part, and uses voice recognition software to convert the speech to text. This data is then input into a generative AI model that determines which part should be installed next and in what order. For example, an instruction might be generated: "Install the piston next." The device receives this instruction, displays an image showing the position of the piston on the display device's screen, and transmits the instruction via voice through the speaker.

[0699] Examples of prompt statements

[0700] "Analyze video and audio of engine parts to determine which part should be installed next and in what order. For example, if a piston needs to be installed next, generate an image showing its location and a voice command saying, 'Install the piston next.'"

[0701] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0702] Step 1:

[0703] The user wears a display device (head-mounted display).

[0704] Specifically, the user wears the display device on his / her head and the display is displayed within the field of vision. The display device has a built-in camera and microphone.

[0705] Step 2:

[0706] The user takes a picture of the engine parts with a camera and inputs into the microphone, "Which part should I install next?"

[0707] Specifically, the user takes a picture of an engine part with the camera, and at the same time speaks into the microphone, "Which part should I install next?" The input data is video data and audio data.

[0708] Step 3:

[0709] The device captures video captured by the camera and audio recorded by the microphone.

[0710] In particular, the terminal receives video data and audio data from the display device, the input data being the video data from the camera and the audio data from the microphone, and the output data being the captured video data and audio data.

[0711] Step 4:

[0712] The video captured by the device is analyzed using image analysis software to identify the type and location of engine parts.

[0713] Specifically, the device uses image analysis software to identify the type and location of engine parts from video data. The input data is the video data, and the output data is the type and location of the parts.

[0714] Step 5:

[0715] The voice captured by the device is converted into text data using voice recognition software.

[0716] Specifically, the terminal uses voice recognition software to convert voice data into text data, with the input data being voice data and the output data being text data.

[0717] Step 6:

[0718] The device inputs the image analysis results and voice recognition results into a generative AI model.

[0719] Specifically, the device inputs the image analysis results (type and position of parts) and voice recognition results (text data) into the generative AI model. The input data are the image analysis results and voice recognition results, and the output data is the input data for the generative AI model.

[0720] Step 7:

[0721] A generative AI model determines which parts to install next and in what order.

[0722] Specifically, the generative AI model analyzes the input data and determines the next part to be installed and the procedure for doing so. The input data is the results of image analysis and voice recognition, and the output data is the next part to be installed and the procedure for doing so.

[0723] Step 8:

[0724] The instructions determined by the generative AI model are output as text data.

[0725] Specifically, the instructions determined by the generative AI model are output to the terminal as text data. The input data is the next part to be installed and the procedure for installing it, and the output data is text data.

[0726] Step 9:

[0727] The terminal receives instructions from the generative AI model and displays an image showing the location of the parts on the display device's screen.

[0728] Specifically, the terminal receives instructions from the generative AI model and displays an image showing the position of the parts on the display of the display device. The input data is text data, and the output data is the image displayed on the display.

[0729] Step 10:

[0730] The device uses speech synthesis software to generate a voice message saying, "Next, install the piston," and communicates this to the user through the speaker.

[0731] In concrete terms, the terminal uses speech synthesis software to generate a voice message saying, "Next, install the piston," and transmits this message to the user through the speaker of the display device. The input data is text data, and the output data is the generated voice message.

[0732] (Application example 2)

[0733] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0734] In conventional assembly and repair work, workers had to spend a lot of time and effort trying to figure out which part to install next and the procedure for doing so. There was also a risk that workers would install the wrong part, which could have a negative impact on work efficiency and quality. This has led to a demand for improved work efficiency and quality.

[0735] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0736] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generating AI, a means for the generating AI to output instructions for the next work procedure to the worker as voice or image, a means for the generating AI to analyze the image and determine the next part to be installed and the procedure, a means for displaying an image showing the position of the part on the head mounted display, and a means for issuing voice instructions through a speaker. This enables the worker to accurately grasp the next part to be installed and the procedure in real time, thereby improving work efficiency and quality.

[0737] "Worker" means a person who assembles or repairs machines or vehicles in a factory or work site.

[0738] A "camera" is a device that captures images and records them as digital data.

[0739] A "microphone" is a device that records sound and saves it as digital data.

[0740] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[0741] "Input" refers to the input of data or information into a system.

[0742] "Output" refers to the system outputting the results of its processing.

[0743] A "means" is a method or device for achieving a specific purpose.

[0744] "Video" refers to visual information captured by a camera.

[0745] "Sound" refers to auditory information recorded by a microphone.

[0746] "Instructions" refers to commands to perform a particular action or task.

[0747] An "image" is digital data that represents visual information.

[0748] A "head-mounted display" is a display device that is worn on the head.

[0749] A "speaker" is a device for reproducing sound.

[0750] A "part" is an individual element that makes up a machine or vehicle.

[0751] A "procedure" is the order or method for performing a particular task.

[0752] As an embodiment of the present invention, we provide a system in which a factory robot is fitted with a head-mounted display and, when performing assembly work, uses generative AI to indicate in real time the next part to be installed and the procedure for doing so.

[0753] The server uses the following hardware and software:

[0754] Hardware: Camera, microphone, head-mounted display, speaker

[0755] Software: OpenCV (video capture and display), pyttsx3 (voice synthesis), transformers (generative AI model)

[0756] The server first uses a camera to capture the current assembly status and sends the video to the generation AI. The generation AI analyzes the video and audio and determines which parts should be installed next and in what order. It then displays an image showing the position of the parts on the head-mounted display and issues a voice command over the speaker, saying, "Install the piston next."

[0757] As a concrete example, when a robot is assembling an engine, a camera captures the current assembly status and sends a voice command such as "Which part should be installed next?" to the generation AI. The generation AI interprets this as "Install the piston next," displays an image showing the piston position on the head-mounted display, and issues a voice command over the speaker.

[0758] An example of a prompt is as follows:

[0759] "Please analyze the video showing the current assembly status and tell me which part should be installed next and in what order."

[0760] In this way, factory robots can efficiently carry out assembly work.

[0761] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0762] Step 1:

[0763] The server uses a camera to capture the current assembly status.

[0764] Input: Video data from the camera

[0765] Output: Captured video frames

[0766] How it works: The camera captures video in real time and acquires data frame by frame.

[0767] Step 2:

[0768] The server sends the captured video frames to the generation AI.

[0769] Input: Captured video frame

[0770] Output: Video data sent to the generation AI

[0771] What it does: Video frames are sent as digital data to a generative AI model.

[0772] Step 3:

[0773] The server uses a microphone to record the voice of the worker.

[0774] Input: Audio data from the microphone

[0775] Output: Recorded audio data

[0776] Specific operation: A microphone records the worker's voice in real time and stores it as digital data.

[0777] Step 4:

[0778] The server sends the recorded voice data to the generation AI.

[0779] Input: Recorded audio data

[0780] Output: Audio data sent to the generation AI

[0781] Specific operations: Audio data is sent as digital data to the generative AI model.

[0782] Step 5:

[0783] The generative AI analyzes video and audio data to determine the next part to be installed and the order in which it should be installed.

[0784] Input: Video data, audio data

[0785] Output: Text data of the next part to be installed and the procedure

[0786] Specific operation: The generative AI model analyzes video and audio and generates the appropriate parts and steps as text.

[0787] Step 6:

[0788] The server displays instructions from the generation AI on a head-mounted display.

[0789] Input: Text data of the next part to be installed and its procedure

[0790] Output: Image data displayed on the head-mounted display

[0791] Specific operation: Text data is converted into image data and displayed on a head-mounted display.

[0792] Step 7:

[0793] The server outputs instructions from the generation AI as audio through a speaker.

[0794] Input: Text data of the next part to be installed and its procedure

[0795] Output: Audio data output from the speaker

[0796] Specific operation: Convert text data into audio data and play it over a speaker.

[0797] In this way, the server instructs the worker on the next work procedure in real time, supporting efficient assembly work.

[0798] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0799] "Example 1"

[0800] In one embodiment of the present invention, a worker wears a head-mounted display. This display has a built-in camera and microphone, and an emotion engine. The worker starts a task, takes an image of the part with the camera of the head-mounted display, and records questions about the task with the microphone. These images and voices are input to the generation AI. The generation AI analyzes the images and voices and determines the next task procedure. Furthermore, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next task procedure based on the emotions. For example, if the emotion engine senses that the worker is stressed, it outputs an instruction to encourage the worker to take a break.

[0801] "Example 2"

[0802] In another embodiment of the present invention, a worker starts assembling an automobile engine, takes images of engine parts with a camera on a head-mounted display, and records a question with a microphone: "Which part should I install next?" These images and voices are input to a generative AI. The generative AI analyzes the images and voices and determines the next part to be installed and the order in which to install it. Furthermore, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on the emotions. For example, if the worker feels tired, the emotion engine outputs an instruction to encourage the worker to take a break.

[0803] The process flow of each embodiment will be described below.

[0804] "Example 1"

[0805] Step 1: The worker wears a head-mounted display, which has a built-in camera, microphone, and emotion engine.

[0806] Step 2: The worker begins the task, taking images of the part with the camera in the head-mounted display and recording questions about the task with the microphone.

[0807] Step 3: These images and audio are input into the generative AI.

[0808] Step 4: The generative AI analyzes the video and audio and determines the next steps.

[0809] Step 5: The emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on those emotions.

[0810] Step 6: For example, if the worker feels stressed, the emotion engine outputs an instruction to encourage the worker to take a break.

[0811] "Example 2"

[0812] Step 1: A worker begins assembling a car engine, taking images of the engine parts with a camera in a head-mounted display and recording a question with a microphone: "Which part should I install next?"

[0813] Step 2: These images and audio are input into the generative AI.

[0814] Step 3: The generative AI analyzes the video and audio to determine the next part to be installed and the order in which it should be installed.

[0815] Step 4: The emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on those emotions.

[0816] Step 5: For example, if a worker feels tired, the emotion engine outputs an instruction to encourage the worker to take a break.

[0817] Example 1

[0818] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0819] In conventional work support systems, it is sometimes difficult to give appropriate instructions on the next work procedure that a worker should perform. In addition, because the system gives uniform instructions without considering the worker's emotional state, it is difficult to reduce the worker's stress and fatigue. This reduces work efficiency and increases the possibility of work errors.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0821] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the voice of the worker and adjust the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate instructions for work procedures that take into account the emotional state of the worker, which is expected to improve work efficiency and reduce work errors.

[0822] "Worker" refers to a person who performs a particular task.

[0823] "Camera" refers to a device for capturing images.

[0824] "Microphone" refers to a device for recording sound.

[0825] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[0826] "Input" refers to entering data into a system.

[0827] "Output" refers to sending data out of the system.

[0828] An "emotion engine" is a system that recognizes emotions from voice and video, and generates appropriate instructions based on that information.

[0829] "Video" refers to visual information captured by a camera.

[0830] "Audio" refers to auditory information recorded by a microphone.

[0831] "Work procedure" refers to the specific steps or methods for performing a particular task.

[0832] "Instructions" refer to information intended to encourage a particular action.

[0833] A "system" refers to a series of devices or programs in which multiple elements work together.

[0834] This invention is a system in which a worker wears a head-mounted display and inputs video and audio to a generative AI via the display's built-in camera and microphone. The generative AI analyzes the video of the parts the worker should handle and the worker's audio (e.g., "Which part should I install next?") and determines the next work procedure. The instructions are communicated to the worker via the display and speaker of the head-mounted display.

[0835] Hardware and Software Used

[0836] Hardware: Head-mounted display (built-in camera, microphone, display, speaker)

[0837] Software: Generative AI models, emotion engines

[0838] Specific operation of the system

[0839] Subject: Server

[0840] The server receives video and audio data transmitted from the head-mounted display worn by the worker. This data is collected through the built-in camera and microphone. The server analyzes the received video data and identifies the part that the worker should handle. It also analyzes the audio data to understand the worker's question (e.g., "Which part should I install next?"). Based on the results of this analysis, the server determines the next work procedure and conveys those instructions to the worker via the display and speakers of the head-mounted display.

[0841] Subject: Terminal

[0842] The terminal (head-mounted display) is a device worn by the worker. The terminal uses a built-in camera and microphone to collect images of parts within the worker's field of vision and the worker's voice. The collected data is sent to the server in real time. When the terminal receives instructions from the server, it displays the instructions on the display and conveys them to the worker by voice through the speaker.

[0843] Subject: User

[0844] The user (worker) wears the head-mounted display while performing their work. As the user handles parts, the built-in camera captures the image, and the microphone records any questions about the work. When the user asks, "Which part should I install next?", the voice data is sent to the server. After receiving instructions from the server, the user checks the next work procedure through the display and speaker of the head-mounted display.

[0845] Examples

[0846] Example 1: Parts installation

[0847] When a user asks, "Which part should I attach next?", the microphone in the head-mounted display records the voice and sends it to the server. The server analyzes the video and audio data to identify the next part to be attached. For example, if the server instructs, "Attach the screw next," the instruction is displayed on the display of the head-mounted display and communicated by voice through the speaker. The user follows the instruction to attach the screw.

[0848] Example 2: Emotion-recognition-based break instruction

[0849] If a user says "I'm tired" while working, the microphone in the head-mounted display records the voice and sends it to the server. The emotion engine recognizes the user's stress from the voice. The emotion engine generates an instruction to encourage the user to take a break, and the server sends the instruction to the head-mounted display. The device displays the instruction "Let's take a short break" on the display and conveys it by voice through the speaker. The user follows the instruction and takes a break.

[0850] Examples of prompt statements

[0851] "If a worker wears a head-mounted display and asks what the next step is, please explain how the generative AI model would determine the next step and communicate it to the worker."

[0852] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0853] Step 1:

[0854] The user wears a head-mounted display.

[0855] Specifically, the user wears the head-mounted display and checks that the built-in camera and microphone are working properly, enabling the collection of video and audio.

[0856] Step 2:

[0857] The device collects video and audio.

[0858] Specifically, the device's built-in camera captures images of parts within the user's field of vision, and the built-in microphone records the user's voice. For example, if the user asks, "Which part should I install next?", that voice is recorded.

[0859] Input: Images of parts in the user's field of vision, user's voice

[0860] Output: Collected video and audio data

[0861] Step 3:

[0862] The terminal transmits the collected data to the server.

[0863] Specifically, the device transmits collected video and audio data in real time to the server, where it is then ready to analyze the data.

[0864] Input: Collected video and audio data

[0865] Output: Data sent to the server

[0866] Step 4:

[0867] The server parses the data.

[0868] Specifically, the server analyzes the received video data to identify the part the user is handling. It also analyzes the voice data to understand the user's question. For example, if the user asks, "Which part should I install next?", the server analyzes and understands the question.

[0869] Input: Video and audio data sent to the server

[0870] Output: Analysis results (part identification, question understanding)

[0871] Step 5:

[0872] The server decides the next step to take.

[0873] Specifically, the server determines the next step based on the analysis results. For example, if the user determines that the part to be installed is a screw, the server generates the instruction "Install the screw next."

[0874] Input: Analysis results (part identification, understanding of question content)

[0875] Output: Next step instructions

[0876] Step 6:

[0877] The server sends the instruction to the terminal.

[0878] Specifically, the server sends instructions for the determined work procedure to the terminal, including the specific work that the user should do next.

[0879] Input: Next step instructions

[0880] Output: Instructions sent to the terminal

[0881] Step 7:

[0882] The terminal conveys the instructions to the user.

[0883] Specifically, the terminal conveys the instructions received from the server to the user. Specifically, the instructions are displayed on the head-mounted display and conveyed by voice through the speaker. For example, the instruction "Next, install the screws" is displayed on the display and conveyed by voice.

[0884] Input: Instructions sent to the terminal

[0885] Output: Instructions given to the user

[0886] (Application example 1)

[0887] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0888] In conventional factory work, workers had to refer to the manual or ask their superiors to confirm the next work procedure, which led to problems of reduced work efficiency. In addition, work instructions were given without considering the emotional state of the workers, which increased the workers' stress and increased the risk of mistakes and accidents.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0890] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the voice of the worker and adjust the instructions for the next work procedure based on the emotions. This allows the worker to not only check the next work procedure in real time, but also to receive appropriate instructions according to his / her emotional state.

[0891] "Worker" refers to the person who actually performs the work in a factory or on-site.

[0892] "Camera" refers to a device that captures images and records them as digital data.

[0893] A "microphone" refers to a device that records sound and saves it as digital data.

[0894] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[0895] An "emotion engine" refers to a system that recognizes emotions from voice and video, and generates appropriate instructions based on the results.

[0896] "Video" refers to visual information captured through a camera.

[0897] "Speech" refers to auditory information captured through a microphone.

[0898] "Means" refers to a method or device for achieving a specific purpose.

[0899] A "system" refers to a set of devices or programs in which multiple elements function in conjunction with one another.

[0900] A "work procedure" refers to the series of steps or instructions required to complete a particular task.

[0901] "Output" refers to the output of results generated by the system to the outside.

[0902] "Input" refers to entering data into a system.

[0903] The system for implementing this invention captures video and audio using a camera and microphone held by the worker, and inputs this to the generation AI. The generation AI analyzes this data and determines the next work procedure. The emotion engine also recognizes emotions from the worker's voice and adjusts instructions based on those emotions.

[0904] Hardware and Software Used

[0905] Hardware: Head-mounted display (HMD), built-in camera, built-in microphone

[0906] Software: OpenCV (camera image acquisition), SpeechRecognition (voice acquisition), EmotionRecognizer (emotion recognition), AIDecisionMaker (generative AI model)

[0907] System Operation

[0908] The server acquires video through the HMD's camera and audio through the microphone. This data is input to the generative AI in real time. The generative AI analyzes the video and audio and determines the next work procedure. For example, if a worker asks, "Which part should I install next?" the generative AI analyzes the video and audio and issues the instruction, "Install the screws next."

[0909] Furthermore, the emotion engine recognizes the emotion of the worker from the voice. For example, if the worker is feeling stressed, the emotion engine will instruct the worker to "take a break." In this way, appropriate instructions are provided according to the worker's emotional state.

[0910] Examples

[0911] Take factory assembly work as an example. A worker wears an HMD, takes pictures of parts with the built-in camera, and records questions about the work with the built-in microphone. This data is input into the generative AI, which determines the next work procedure. For example, if a worker asks, "Which part should I attach next?" the generative AI will instruct, "Attach the screws next." If the worker is feeling stressed, the emotion engine will instruct, "Take a break."

[0912] Examples of prompt statements

[0913] An example of a prompt sentence to be input to the generative AI model is as follows.

[0914] The worker is asking, "Which part should I install next?" The video shows parts A and B. Please tell me the next step.

[0915] By inputting this prompt into a generative AI model, the next steps can be determined.

[0916] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0917] Step 1:

[0918] The server acquires images through the HMD camera. The image data from the camera is captured in real time and stored as digital data. This image data includes visual information of the parts handled by the worker and the work environment.

[0919] Step 2:

[0920] The server receives voice from the microphone of the HMD. The voice data from the microphone is captured in real time and stored as digital data. This voice data includes the worker's questions and instructions.

[0921] Step 3:

[0922] The server inputs the acquired video and audio data into the generation AI, which then analyzes the data and determines the next work procedure. Specifically, it identifies the type and location of parts from the video data, and understands the content of the worker's questions from the audio data.

[0923] Step 4:

[0924] The generative AI determines the next work procedure based on the analysis results and generates the instructions. For example, if a worker asks, "Which part should I attach next?", the generative AI generates the instruction, "Attach the screws next."

[0925] Step 5:

[0926] The server outputs instructions from the generation AI to the worker through the HMD's display and speakers. Visual instructions are displayed on the display, and audio instructions are played back from the speakers.

[0927] Step 6:

[0928] The server uses an emotion engine to recognize the worker's emotions from the voice data. The emotion engine analyzes the tone, speed, and content of the voice to determine whether the worker is feeling stressed.

[0929] Step 7:

[0930] The emotion engine recognizes the worker's emotions and adjusts the instructions of the generative AI based on the results. For example, if a worker is feeling stressed, the emotion engine will generate the instruction "Please take a break."

[0931] Step 8:

[0932] The server outputs the adjusted instructions to the worker again through the HMD display and speakers, allowing the worker to receive appropriate instructions according to their emotional state.

[0933] Example 2

[0934] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0935] Conventional work support systems have difficulty instructing workers on the next steps, and are particularly unable to provide instructions that take into account the worker's emotions and fatigue state. This leads to problems such as reduced work efficiency and increased worker fatigue and stress.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0937] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for recognizing emotions from the voice of the worker using an emotion engine and adjusting the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate instructions for work procedures that take into account the emotions and fatigue state of the worker.

[0938] "Worker" refers to a person who performs work such as assembling or repairing machines, vehicles, etc.

[0939] "Camera" refers to a device for capturing images.

[0940] "Microphone" refers to a device for recording sound.

[0941] "Generative AI" refers to an artificial intelligence model that analyzes input data and generates an appropriate output.

[0942] An "emotion engine" refers to technology for recognizing emotions from data such as audio and video.

[0943] "Video" refers to visual information captured through a camera.

[0944] "Speech" refers to auditory information captured through a microphone.

[0945] "Input" refers to entering data into a system.

[0946] "Output" refers to sending data out of the system.

[0947] "Means" refers to a method or device for achieving a specific purpose.

[0948] "Work procedure" refers to the specific steps and methods for proceeding with work.

[0949] "Instructions" refers to communicating to workers the actions and procedures they should follow.

[0950] "Image" refers to a still image or video used to express visual information.

[0951] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0952] This invention is a system that uses a head-mounted display (HMD) to instruct workers on the next part to be installed and the procedure for installing it when assembling or repairing machines, vehicles, etc. The system is composed of a generative AI model that performs video and audio analysis, and an emotion engine that recognizes emotions.

[0953] Hardware and Software Used

[0954] Head-mounted display (HMD): Equipped with a camera and microphone, it captures the worker's visual and auditory information.

[0955] Server: Provides the computational resources to run the generative AI model and the emotion engine.

[0956] Generative AI model: Analyzes video and audio data to determine which part to install next and in what order.

[0957] Emotion engine: Recognizes emotions from a worker's voice and adjusts instructions for the next work step based on those emotions.

[0958] System Operation

[0959] 1. The user puts on the HMD:

[0960] The user puts on the HMD and starts assembling engine parts. The HMD has a built-in camera and microphone, which are used to record the progress of the work.

[0961] 2. The device acquires video and audio:

[0962] The device (HMD) uses a camera to capture images of engine parts in real time and a microphone to record the user's voice, for example when the user asks, "Which part should I install next?"

[0963] 3. The device sends the data to the server:

[0964] The device transmits captured video and recorded audio data to a server, where it is used for analysis by generative AI models and emotion engines.

[0965] 4. The server analyzes the data:

[0966] The server inputs the received video and audio data into a generative AI model. The generative AI model analyzes the state of the engine parts from the video and understands the user's question from the audio. For example, it recognizes from the video that the piston has not yet been installed, and understands from the audio that the user is asking about the next part to be installed.

[0967] 5. The server recognizes emotions:

[0968] The server uses an emotion engine to recognize emotions from the user's voice, for example if the user's voice sounds tired.

[0969] 6. The server generates the instruction:

[0970] The server determines the next part to be installed and the procedure based on the analysis results of the generative AI model and the recognition results of the emotion engine. For example, it generates an instruction such as "Install the piston next." If it recognizes that the user is tired, it also adds an instruction such as "Take a short break."

[0971] 7. The terminal will output the command:

[0972] The terminal receives instructions from the server, displays them on the HMD display as an image showing the position of the parts, and outputs voice instructions from the speaker such as "Next, attach the piston." If the user is tired, it also outputs voice instructions such as "Please take a short break."

[0973] Examples

[0974] Let's say a user is assembling an engine. The HMD camera captures video of the engine parts and asks, "Which part should I install next?" through the microphone. The device sends this data to the server. The server uses a generative AI model to analyze the video and audio, and generates the instruction, "Install the piston next." Furthermore, if the emotion engine recognizes fatigue from the user's voice, it also adds the instruction, "Please take a short break." The device communicates these instructions to the user through the HMD's display and speaker.

[0975] Examples of prompt statements

[0976] "We are currently assembling the engine. What is the next part to be installed and in what order? Please analyze the current video and audio data."

[0977] This system combines an HMD, a generative AI model, and an emotion engine to improve the user's work efficiency and provide flexible instructions according to the worker's condition.

[0978] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0979] Step 1:

[0980] The user puts on the HMD and starts assembling engine parts. The user uses the camera and microphone of the HMD to record the progress of the work. Specifically, the user asks, "Which part should I install next?"

[0981] Input: User's audio and video

[0982] Output: Audio and video data recorded in the HMD

[0983] Step 2:

[0984] The device (HMD) captures images of engine parts and records audio. The camera captures images in real time, and the microphone records the user's questions. For example, the device captures images of engine parts and audio such as "Which part should I install next?"

[0985] Input: User's audio and video

[0986] Output: Shooting video data and recording audio data

[0987] Step 3:

[0988] The device transmits video and audio data to the server. The HMD transmits the captured data to the server, where it is used for analysis by the generative AI model and emotion engine.

[0989] Input: Recorded video data and recorded audio data

[0990] Output: Video and audio data sent to the server

[0991] Step 4:

[0992] The server analyzes the video and audio using a generative AI model. The server inputs the received video and audio data into the generative AI model, analyzes the state of the engine parts from the video, and understands the user's question from the audio. For example, it recognizes from the video that the piston has not yet been installed, and understands from the audio that the user is asking about the next part to be installed.

[0993] Input: Video and audio data sent to the server

[0994] Output: Analyzed part status and user question

[0995] Step 5:

[0996] The server uses the emotion engine to recognize the user's emotion. The server uses the emotion engine to recognize the emotion from the user's voice. For example, if the user's voice sounds tired, the server recognizes that emotion.

[0997] Input: User's voice data

[0998] Output: The perceived emotional state of the user.

[0999] Step 6:

[1000] The server generates instructions based on the analysis results and emotion recognition results. Based on the analysis results of the generative AI model and the recognition results of the emotion engine, the server determines the next part to be installed and the procedure for doing so. For example, it generates an instruction such as "Install the piston next." If it recognizes that the user is tired, it also adds an instruction such as "Take a short break."

[1001] Input: Analyzed part states and the user's emotional state

[1002] Output: Generated work instructions

[1003] Step 7:

[1004] The terminal displays and speaks instructions to the user. The terminal receives instructions from the server and displays them on the HMD display as an image showing the positions of parts. It also outputs voice instructions from the speaker, such as "Next, attach the piston." If the user is tired, it also outputs voice instructions such as "Please take a short break."

[1005] Input: Generated work order

[1006] Output: Images displayed on the HMD display and audio instructions output from the speaker.

[1007] (Application example 2)

[1008] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1009] Conventional assembly work support systems have difficulty instructing workers on the next work procedure, which reduces work efficiency. In addition, they are unable to give instructions that take into account the worker's fatigue and stress, which can have a negative impact on the worker's health and work quality. To solve these problems, a system that recognizes the worker's emotions and issues appropriate instructions based on them is needed.

[1010] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1011] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the worker's voice and adjust the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate work instructions that take into account the emotions of the worker, improving work efficiency and maintaining the health of the worker.

[1012] A "camera" is a device for capturing images.

[1013] A "microphone" is a device for capturing sound.

[1014] "Generative AI" is artificial intelligence that analyzes input data and generates appropriate output.

[1015] An "emotion engine" is a system that recognizes emotions from data such as audio and video, and generates appropriate instructions based on those emotions.

[1016] A "task procedure" is the sequence of steps required to complete a particular task.

[1017] "Voice" refers to words and sounds made by the worker.

[1018] "Video" is visual information captured through a camera.

[1019] "Output" refers to the provision of instructions and information generated by the system to workers.

[1020] "Input" refers to data or information entered into a system.

[1021] A "system" is a configuration in which multiple devices and software work together to achieve a specific function.

[1022] The system for implementing this invention captures video and audio using a camera and microphone held by the worker, and inputs them into the generation AI. The generation AI analyzes the next work procedure and outputs it to the worker as audio or images. The emotion engine also recognizes emotions from the worker's voice and adjusts the instructions for the next work procedure based on those emotions.

[1023] Hardware and Software Used

[1024] Hardware:

[1025] Camera (e.g. Logitech C920)

[1026] Microphone (e.g. Blue Yeti)

[1027] Head-mounted displays (e.g. Microsoft HoloLens)

[1028] software:

[1029] OpenCV (for image capture)

[1030] SpeechRecognition (for voice capture)

[1031] Transformers (using generative AI and sentiment analysis models)

[1032] Data processing and calculation

[1033] The server receives the video captured through the camera and the audio captured through the microphone. These data are input to the generative AI model, which generates the next steps to be taken. The generated instructions are displayed on the head-mounted display or communicated to the worker by voice. In addition, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions based on those emotions.

[1034] Examples

[1035] When a worker asks, "Which part should I install next?", a camera takes a picture of the engine part and inputs it into the generative AI model. The generative AI model generates an instruction, such as "Install the piston next," and displays it on the head-mounted display. Furthermore, if the emotion engine senses fatigue from the worker's voice, it issues the instruction, "Please take a break."

[1036] Examples of prompt statements

[1037] Image: <Engine parts image>

[1038] Q: What part should I install next?

[1039] Please tell me what parts I should install next and what steps to take.

[1040] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1041] Step 1:

[1042] A user uses a camera to capture video of an engine part.

[1043] Input: Physical condition of engine parts

[1044] Output: Captured image data of engine parts

[1045] Specific operation: A camera (e.g. Logitech C920) is used to capture images of engine parts. The image data is sent to the server in real time.

[1046] Step 2:

[1047] The user uses a microphone to capture audio saying, "Which part should I install next?"

[1048] Input: User's voice

[1049] Output: Captured audio data

[1050] What it does: Use a microphone (e.g. Blue Yeti) to record the user's question. The audio data is sent to the server in real time.

[1051] Step 3:

[1052] The server inputs the captured video and audio into a generative AI model.

[1053] Input: Video data, audio data

[1054] Output: Input data for the generative AI model

[1055] How it works: The server receives the captured video and audio and formats the data for input into a generative AI model (e.g. GPT-3.5-turbo).

[1056] Step 4:

[1057] The generative AI model analyzes the next work step and generates instructions.

[1058] Input: Input data (video data, audio data)

[1059] Output: Work procedure instructions

[1060] Specific operation: The generative AI model analyzes the input data and determines which parts to install next and in what order, for example generating instructions such as "Install the piston next."

[1061] Step 5:

[1062] The server displays the generated instructions on a head-mounted display or conveys them to the user by voice.

[1063] Input: Work procedure instructions

[1064] Output: Display or audio instructions

[1065] Specific operation: The server displays the generated instructions on the display of a head-mounted display (e.g., Microsoft HoloLens) or conveys them to the user audibly through the speakers.

[1066] Step 6:

[1067] The server uses an emotion engine to recognize emotions from the user's voice.

[1068] Input: User's voice data

[1069] Output: Emotion recognition result

[1070] Specific operation: The emotion engine analyzes the user's voice data and recognizes the user's emotions (e.g., tiredness, stress).

[1071] Step 7:

[1072] The server adjusts the instructions for the next work step based on the emotion.

[1073] Input: Emotion recognition results

[1074] Output: Coordinated work instructions

[1075] Specific Actions: Based on the results of the emotion engine, the server adjusts the next steps of the work. For example, if the server detects that the user is tired, it will give the instruction "Please take a break."

[1076] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1077] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1078] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1079] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1080] [Third embodiment]

[1081] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1082] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1083] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1084] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1085] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1086] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1087] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1088] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1089] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1090] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1091] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1092] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1093] "Example 1"

[1094] In one embodiment of the present invention, a worker wears a head-mounted display and inputs video and audio to the generation AI through the display's built-in camera and microphone. The generation AI analyzes the video of the parts the worker should handle and the worker's audio (e.g., "Which part should I install next?") and determines the next work procedure. The instructions are conveyed to the worker through the display and speaker of the head-mounted display.

[1095] "Example 2"

[1096] As a specific example, consider the assembly of an automobile engine. A worker wears a head-mounted display and inputs images of engine parts and the voice saying, "Which part should I install next?" into the generative AI. The generative AI analyzes the video and voice and determines the next part to be installed and the order in which to install it. The instructions are communicated to the worker as an image showing the position of the parts on the head-mounted display screen and as a voice message over the speaker saying, "Install the piston next."

[1097] The process flow of each embodiment will be described below.

[1098] "Example 1"

[1099] Step 1: The worker wears a head-mounted display, which has a built-in camera and microphone.

[1100] Step 2: The worker starts the task, takes a picture of the part with the camera in the head-mounted display, and records a question about the task with the microphone. These images and voices are input to the generative AI.

[1101] Step 3: The generative AI analyzes the video and audio and determines the next steps to take.

[1102] Step 4: The generating AI communicates instructions for the next work procedure to the worker via the head-mounted display and speakers.

[1103] "Example 2"

[1104] Step 1: A worker starts assembling a car engine, takes pictures of the engine parts with a camera on a head-mounted display, and uses a microphone to record a question such as, "Which part should I install next?" These images and voices are input to the generative AI.

[1105] Step 2: The generative AI analyzes the video and audio to determine the next part to be installed and the order in which it should be installed.

[1106] Step 3: The generating AI communicates to the worker the position of the parts in the form of an image on the head-mounted display and audibly through the speaker, "Next, install the piston."

[1107] Example 1

[1108] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1109] Conventional work assistance systems have had difficulty efficiently understanding the next work procedure that a worker should perform. In addition, they were unable to analyze data collected through the camera and microphone carried by the worker in real time and provide appropriate instructions. This resulted in issues such as reduced work efficiency and an increased risk of mistakes.

[1110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1111] In this invention, the server includes a means for inputting video and audio through a display device worn by a worker to the generative AI model, a means for the generative AI model to output instructions for the next work procedure to the worker by audio or image, a means for the display device to transmit collected data to the server, a means for the server to transmit the results of the generative AI model to the display device, and a means for the display device to present the work procedure to the worker. This allows the worker to grasp the next work procedure in real time, improving work efficiency and reducing the occurrence of errors.

[1112] A "worker" is a person who wears a display device and uses the system to perform a specific task.

[1113] A "display device" is a device such as a head-mounted display worn by a worker, which has the function of collecting video and audio and displaying instructions.

[1114] A "generative AI model" is an artificial intelligence model that analyzes collected video and audio data and determines the next work steps.

[1115] The "server" is a computer system whose role is to receive data sent from the display device, input it into the generative AI model, and send the results to the display device.

[1116] "Image" refers to visual information of the work environment and parts captured through the display device's camera.

[1117] "Voice" refers to acoustic information of the worker's comments and questions collected through the microphone of the display device.

[1118] "Input means" refers to a method or device for inputting the video and audio collected by the display device into the generative AI model.

[1119] "Means for output" refers to a method or device for providing the work procedures determined by the generative AI model to workers in the form of audio or images.

[1120] "Transmitting means" refers to a method or apparatus for the display device to transmit collected data to the server and for the server to transmit the results of the generative AI model to the display device.

[1121] "Presenting means" means a method or device by which a display device presents the results of the generative AI model to a human operator visually or audibly.

[1122] This invention is a system that collects video and audio through a display device worn by a worker and inputs them into a generative AI model to provide the next work procedure in real time. A specific embodiment of this system will be described below.

[1123] First, the user wears a head-mounted display (HMD). This HMD has a built-in camera and microphone, and has the function of collecting video and audio of the work environment. The HMD can be a commercially available device such as Microsoft HoloLens or Oculus Rift.

[1124] The device (HMD) uses a camera to capture images of the parts and work environment within the user's field of vision, and records the user's comments and questions with a microphone. This data is sent to a server in real time. Data is sent using wireless communication technologies such as Wi-Fi and Bluetooth.

[1125] The server inputs the received video and audio data into a generative AI model, which is built using frameworks such as TensorFlow and PyTorch. The generative AI model analyzes the video data to identify the part the user is handling, and analyzes the audio data to understand the user's question.

[1126] Once the generative AI model has determined the next work procedure, the result is sent from the server to the device. The device then presents the received work procedure to the user. Specifically, the HMD display shows an image and explanation of the next part to be installed, and the speaker issues audio instructions.

[1127] As a concrete example, consider the case where a user is assembling a car engine. The user is wearing an HMD and handling engine parts. When the user asks, "Which part should I install next?", the HMD's camera captures the current work situation and the microphone records the question. This data is sent to the server, where it is analyzed by the generative AI model. The generative AI model determines that the next part to be installed is a "piston" and returns this information to the server. The server sends the result to the terminal, which displays "Install the piston next" on the HMD's display and also plays the same instruction over the speaker.

[1128] Example prompt:

[1129] A worker is assembling an engine. Capture the current work status with a camera, analyze the worker's question "Which part should I install next?", and instruct the worker on which part to install next.

[1130] In this way, the system allows the server, terminals, and users to work together to efficiently carry out tasks.

[1131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1132] Step 1:

[1133] The user puts on the HMD and begins working.

[1134] Specifically, the user puts on the HMD and turns it on. This activates the HMD's camera and microphone, allowing video and audio to be collected. The input is the video and audio of the parts and work environment within the user's field of vision, and the output is the data collected through the HMD's camera and microphone.

[1135] Step 2:

[1136] The terminal (HMD) collects video and audio.

[1137] Specifically, the HMD camera captures images of the parts and working environment within the user's field of vision, and the microphone records the user's comments and questions. The input is the images and audio of the parts and working environment within the user's field of vision, and the output is the collected video and audio data.

[1138] Step 3:

[1139] The terminal transmits the collected data to the server.

[1140] Specifically, the HMD compresses the collected video and audio data and transmits it to the server using wireless communication technologies such as Wi-Fi and Bluetooth. The input is the collected video and audio data, and the output is the data transmitted to the server.

[1141] Step 4:

[1142] The server inputs the data into a generative AI model.

[1143] Specifically, the server inputs the received video and audio data into a generative AI model. The generative AI model is built using frameworks such as TensorFlow and PyTorch. The input is the video and audio data sent to the server, and the output is the data input into the generative AI model.

[1144] Step 5:

[1145] A generative AI model analyzes the data and determines the next steps to take.

[1146] Specifically, the generative AI model analyzes video data to identify the part the user is handling, and analyzes audio data to understand the user's question. For example, if a user asks, "Which part should I install next?", the generative AI model analyzes the question and determines which part should be installed next. The input is the video and audio data input to the generative AI model, and the output is instructions for the next work procedure.

[1147] Step 6:

[1148] The server sends the results of the generated AI model to the device.

[1149] Specifically, the server sends the next work procedure determined by the generative AI model to the terminal. The data is compressed again and sent to the terminal via the network. The input is the result of the generative AI model, and the output is the work procedure instructions sent to the terminal.

[1150] Step 7:

[1151] The terminal presents the work procedure to the user.

[1152] Specifically, the terminal presents the received work procedure to the user. An image and explanation of the next part to be installed is shown on the HMD display, and instructions are given by voice from the speaker. For example, instructions such as "Install the piston next" are displayed. The input is the work procedure instructions sent to the terminal, and the output is the work procedure presented to the user.

[1153] (Application example 1)

[1154] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1155] In conventional assembly and repair work in factories, workers have to refer to the manual to confirm the next step, which reduces work efficiency. There is also a risk that the quality of work will decline if the worker makes a mistake in the next step. To solve these problems, a system that can provide instructions on work procedures in real time is needed.

[1156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1157] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, a means for the generation AI to analyze the voice command and image of the worker and determine the next work procedure, and a means for communicating the determined work procedure to the worker through the display and speaker of the head mounted display. This allows the worker to check the next work procedure in real time, making it possible to improve work efficiency and maintain work quality.

[1158] "Worker" refers to a person who assembles or repairs machines or parts in a factory or work site.

[1159] "Camera" refers to a device used to capture video and input the data into generative AI.

[1160] A "microphone" refers to a device for recording audio and inputting the data into the generative AI.

[1161] "Generative AI" refers to artificial intelligence that analyzes input video and audio and determines the next steps in the process.

[1162] A "head-mounted display" refers to a device worn by a worker to display and play back images and audio.

[1163] "Display" refers to a screen for visually displaying the work procedures determined by the generation AI.

[1164] "Speaker" refers to a device for playing back the work procedures determined by the generation AI in audio form.

[1165] "Voice command" refers to a voice instruction given by a worker to inquire about the next work procedure.

[1166] "Video" refers to visual information captured through a camera.

[1167] "Work procedure" refers to the specific operation or process that a worker should perform next.

[1168] The system for implementing this invention uses a camera and microphone held by a worker to input video and audio to a generation AI, which then determines the next work procedure and issues instructions to the worker. This system uses the following hardware and software.

[1169] Hardware

[1170] 1. Camera: The camera carried by the worker is used to capture footage of the work site and input the data into the generative AI.

[1171] 2. Microphone: The microphone held by the worker is used to record the worker's voice and input the data into the generative AI.

[1172] 3. Head-Mounted Display (HMD): The HMD worn by the worker is equipped with a display that visually displays the work procedures determined by the generation AI, and speakers that play them audibly.

[1173] software

[1174] 1. OpenCV: Used to acquire and process camera images.

[1175] 2. SpeechRecognition: Used to convert audio recorded from a microphone into text.

[1176] 3. Transformers: Uses generative AI models (e.g. GPT-3) to analyze input video and audio and determine the next steps.

[1177] System Operation

[1178] 1. Acquiring video and audio: The video of the work site and the voice of the worker are acquired through the camera and microphone held by the worker. The camera acquires video data using OpenCV, and the microphone converts the voice data into text using SpeechRecognition.

[1179] 2. Input to the generative AI: The captured video and audio data are input into the generative AI model, which uses Transformers to analyze the video and audio commands and determine the next steps.

[1180] 3. Output of work procedures: The work procedures determined by the generation AI are visually displayed on the HMD display and played audibly through the speaker.

[1181] Examples

[1182] For example, if a worker on a factory assembly line asks, "Which part should I install next?", a camera captures video of the work site and a microphone records the audio. The generative AI model uses the following prompt to determine the next step:

[1183] Example prompt:

[1184] Image: [Part image data], Voice command: "Which part should I install next?" What's the next step?

[1185] The generative AI model analyzes the prompt and determines specific work steps, such as "Next, install the screw." These instructions are displayed on the HMD display and communicated to the worker through the speaker.

[1186] In this way, workers can check the next work procedure in real time, improving work efficiency and maintaining work quality.

[1187] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1188] Step 1:

[1189] The server obtains images from the camera held by the worker. The camera captures images of the work site in real time and processes the data using OpenCV. The input is the image data from the camera, and the output is the processed image data.

[1190] Step 2:

[1191] The server captures the voice from the worker's microphone. The microphone records the worker's voice in real time and converts the data into text using SpeechRecognition. The input is the voice data from the microphone, and the output is the converted text data.

[1192] Step 3:

[1193] The server inputs the acquired video data and text data into the generative AI model. The generative AI model uses Transformers to analyze the video and voice commands and determine the next work steps. The input is video data and text data, and the output is text data regarding the next work steps.

[1194] Step 4:

[1195] The server displays the next work procedure obtained from the generative AI model on the display of a head-mounted display (HMD). The display visually shows the specific operation or process that the worker should perform next. The input is text data regarding the next work procedure, and the output is visual information displayed on the display.

[1196] Step 5:

[1197] The server plays the next work procedure obtained from the generative AI model aloud through the speaker of the head-mounted display (HMD). The speaker aloud tells the worker the specific operation or process that he or she should perform next. The input is text data regarding the next work procedure, and the output is audio information played from the speaker.

[1198] Step 6:

[1199] The user confirms the next steps through the HMD's display and speakers, and performs the work according to the instructions. The input is visual and audio information from the display and speakers, and the output is the work that the worker actually performs.

[1200] Example 2

[1201] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1202] With conventional work support systems, it was difficult to efficiently instruct workers on the next step in the work process. In addition, workers had to stop working to confirm the next step, which reduced work efficiency. Furthermore, it was difficult for workers to accurately grasp the location of parts, which meant there was a risk of installing the wrong parts.

[1203] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1204] In this invention, the server includes a means for attaching a display device held by a worker, a means for acquiring an image through a camera and a voice of the worker through a microphone, a means for analyzing the acquired image with image analysis software and identifying the type and position of parts, a means for converting the acquired voice into text data with voice recognition software, a means for inputting the image analysis result and the voice recognition result into a generative AI model, a means for the generative AI model to determine the next part to be installed and its procedure, a means for outputting the instructions determined by the generative AI model as text data, a means for displaying an image showing the position of the parts on the display of the display device, and a means for generating instructions by voice using voice synthesis software and communicating them to the worker through a speaker. This allows the worker to check the next work procedure without stopping his / her work and to accurately grasp the position of the parts.

[1205] A "display device" is a device worn by a worker to provide visual information.

[1206] A "camera" is a device for taking images and acquiring the data.

[1207] A "microphone" is a device for recording sound and acquiring the data.

[1208] "Image analysis software" is software for analyzing acquired video data and extracting specific information.

[1209] "Voice recognition software" refers to software for converting acquired voice data into text data.

[1210] A "generative AI model" is an artificial intelligence model that analyzes input data and determines the next work procedure to be performed.

[1211] "Text data" is character information converted by voice recognition software.

[1212] A "display" is a screen built into a display device for displaying visual information.

[1213] "Speech synthesis software" means software for converting text data into speech and generating spoken instructions.

[1214] A "speaker" is a device for outputting speech generated by speech synthesis software.

[1215] This invention is a system in which a worker wears a display device, analyzes video and audio captured through a camera and microphone, and uses a generative AI model to instruct the next work procedure. A specific embodiment of this system will be described below.

[1216] Hardware and Software Used

[1217] Hardware: Display device (head-mounted display), camera, microphone, speaker

[1218] Software: Image analysis software, voice recognition software, generative AI models, voice synthesis software

[1219] System Operation

[1220] 1. The user wears a display device (head-mounted display) that has a built-in camera and microphone.

[1221] 2. The user takes a video of the engine parts with a camera and inputs into the microphone, "Which part should I install next?"

[1222] 3. The device captures the video captured by the camera and the audio recorded by the microphone.

[1223] 4. The video captured by the device is analyzed using image analysis software to identify the type and location of the engine parts.

[1224] 5. The voice captured by the device is converted into text data using voice recognition software.

[1225] 6. The device inputs the image analysis results and voice recognition results into the generative AI model.

[1226] 7. The generative AI model determines which part to install next and in what order.

[1227] 8. The instructions determined by the generative AI model are output as text data.

[1228] 9. The terminal receives instructions from the generative AI model and displays an image showing the location of the parts on the display device.

[1229] 10. The device uses speech synthesis software to generate a voice message saying, "Next, install the piston," and communicates this to the user through the speaker.

[1230] Examples

[1231] For example, say a user is assembling an engine. The user puts on the display device, takes pictures of the engine parts with the camera, and speaks into the microphone saying, "Which part should I install next?" The device captures the video and audio, uses image analysis software to identify the type and location of the part, and uses voice recognition software to convert the speech to text. This data is then input into a generative AI model that determines which part should be installed next and in what order. For example, an instruction might be generated: "Install the piston next." The device receives this instruction, displays an image showing the position of the piston on the display device's screen, and transmits the instruction via voice through the speaker.

[1232] Examples of prompt statements

[1233] "Analyze video and audio of engine parts to determine which part should be installed next and in what order. For example, if a piston needs to be installed next, generate an image showing its location and a voice command saying, 'Install the piston next.'"

[1234] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1235] Step 1:

[1236] The user wears a display device (head-mounted display).

[1237] Specifically, the user wears the display device on his / her head and the display is displayed within the field of vision. The display device has a built-in camera and microphone.

[1238] Step 2:

[1239] The user takes a picture of the engine parts with a camera and inputs into the microphone, "Which part should I install next?"

[1240] Specifically, the user points the camera at an engine part and takes a picture of it, while at the same time speaking into the microphone, "Which part should I install next?" The input data is video data and audio data.

[1241] Step 3:

[1242] The device captures video captured by the camera and audio recorded by the microphone.

[1243] In a specific operation, the terminal receives video data and audio data from the display device, the input data being the video data from the camera and the audio data from the microphone, and the output data being the captured video data and audio data.

[1244] Step 4:

[1245] The video captured by the device is analyzed using image analysis software to identify the type and location of engine parts.

[1246] Specifically, the device uses image analysis software to identify the type and location of engine parts from video data. The input data is the video data, and the output data is the type and location of the parts.

[1247] Step 5:

[1248] The voice captured by the device is converted into text data using voice recognition software.

[1249] Specifically, the terminal uses voice recognition software to convert voice data into text data, with the input data being voice data and the output data being text data.

[1250] Step 6:

[1251] The device inputs the image analysis results and voice recognition results into a generative AI model.

[1252] Specifically, the device inputs the image analysis results (type and position of parts) and voice recognition results (text data) into the generative AI model. The input data are the image analysis results and voice recognition results, and the output data is the input data for the generative AI model.

[1253] Step 7:

[1254] A generative AI model determines which parts to install next and in what order.

[1255] Specifically, the generative AI model analyzes the input data and determines the next part to be installed and the procedure for doing so. The input data is the results of image analysis and voice recognition, and the output data is the next part to be installed and the procedure for doing so.

[1256] Step 8:

[1257] The instructions determined by the generative AI model are output as text data.

[1258] Specifically, the instructions determined by the generative AI model are output to the terminal as text data. The input data is the next part to be installed and the procedure for installing it, and the output data is text data.

[1259] Step 9:

[1260] The terminal receives instructions from the generative AI model and displays an image showing the location of the parts on the display device's screen.

[1261] Specifically, the terminal receives instructions from the generative AI model and displays an image showing the position of the parts on the display of the display device. The input data is text data, and the output data is the image displayed on the display.

[1262] Step 10:

[1263] The device uses speech synthesis software to generate a voice message saying, "Next, install the piston," and communicates this to the user through the speaker.

[1264] In concrete terms, the terminal uses speech synthesis software to generate a voice message saying, "Next, install the piston," and transmits this message to the user through the speaker of the display device. The input data is text data, and the output data is the generated voice message.

[1265] (Application example 2)

[1266] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1267] In conventional assembly and repair work, workers had to spend a lot of time and effort trying to figure out which part to install next and the procedure for doing so. There was also a risk that workers would install the wrong part, which could have a negative impact on work efficiency and quality. This has led to a demand for improved work efficiency and quality.

[1268] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1269] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generating AI, a means for the generating AI to output instructions for the next work procedure to the worker as voice or image, a means for the generating AI to analyze the image and determine the next part to be installed and the procedure, a means for displaying an image showing the position of the part on the head mounted display, and a means for issuing voice instructions through a speaker. This enables the worker to accurately grasp the next part to be installed and the procedure in real time, thereby improving work efficiency and quality.

[1270] "Worker" means a person who assembles or repairs machines or vehicles in a factory or work site.

[1271] A "camera" is a device that captures images and records them as digital data.

[1272] A "microphone" is a device that records sound and saves it as digital data.

[1273] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[1274] "Input" refers to the input of data or information into a system.

[1275] "Output" refers to the system outputting the results of its processing.

[1276] A "means" is a method or device for achieving a specific purpose.

[1277] "Video" refers to visual information captured by a camera.

[1278] "Sound" refers to auditory information recorded by a microphone.

[1279] "Instructions" refers to commands to perform a particular action or task.

[1280] An "image" is digital data that represents visual information.

[1281] A "head-mounted display" is a display device that is worn on the head.

[1282] A "speaker" is a device for reproducing sound.

[1283] A "part" is an individual element that makes up a machine or vehicle.

[1284] A "procedure" is the order or method for performing a particular task.

[1285] As an embodiment of the present invention, we provide a system in which a factory robot is fitted with a head-mounted display and, when performing assembly work, uses generative AI to indicate in real time the next part to be installed and the procedure for doing so.

[1286] The server uses the following hardware and software:

[1287] Hardware: Camera, microphone, head-mounted display, speaker

[1288] Software: OpenCV (video capture and display), pyttsx3 (voice synthesis), transformers (generative AI model)

[1289] The server first uses a camera to capture the current assembly status and sends the video to the generation AI. The generation AI analyzes the video and audio and determines which parts should be installed next and in what order. It then displays an image showing the position of the parts on the head-mounted display and issues a voice command over the speaker, saying, "Install the piston next."

[1290] As a concrete example, when a robot is assembling an engine, a camera captures the current assembly status and sends a voice command such as "Which part should be installed next?" to the generation AI. The generation AI interprets this as "Install the piston next," displays an image showing the piston position on the head-mounted display, and issues a voice command over the speaker.

[1291] An example of a prompt is as follows:

[1292] "Please analyze the video showing the current assembly status and tell me which part should be installed next and in what order."

[1293] In this way, factory robots can efficiently carry out assembly work.

[1294] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1295] Step 1:

[1296] The server uses a camera to capture the current assembly status.

[1297] Input: Video data from the camera

[1298] Output: Captured video frames

[1299] How it works: The camera captures video in real time and acquires data frame by frame.

[1300] Step 2:

[1301] The server sends the captured video frames to the generation AI.

[1302] Input: Captured video frame

[1303] Output: Video data sent to the generation AI

[1304] What it does: Video frames are sent as digital data to a generative AI model.

[1305] Step 3:

[1306] The server uses a microphone to record the voice of the worker.

[1307] Input: Audio data from the microphone

[1308] Output: Recorded audio data

[1309] Specific operation: A microphone records the worker's voice in real time and stores it as digital data.

[1310] Step 4:

[1311] The server sends the recorded voice data to the generation AI.

[1312] Input: Recorded audio data

[1313] Output: Audio data sent to the generation AI

[1314] Specific operations: Audio data is sent as digital data to the generative AI model.

[1315] Step 5:

[1316] The generative AI analyzes video and audio data to determine the next part to be installed and the order in which it should be installed.

[1317] Input: Video data, audio data

[1318] Output: Text data of the next part to be installed and the procedure

[1319] Specific operation: The generative AI model analyzes video and audio and generates the appropriate parts and steps as text.

[1320] Step 6:

[1321] The server displays instructions from the generation AI on a head-mounted display.

[1322] Input: Text data of the next part to be installed and its procedure

[1323] Output: Image data displayed on the head-mounted display

[1324] Specific operation: Text data is converted into image data and displayed on a head-mounted display.

[1325] Step 7:

[1326] The server outputs instructions from the generation AI as audio through a speaker.

[1327] Input: Text data of the next part to be installed and its procedure

[1328] Output: Audio data output from the speaker

[1329] Specific operation: Convert text data into audio data and play it over a speaker.

[1330] In this way, the server instructs the worker on the next work procedure in real time, supporting efficient assembly work.

[1331] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1332] "Example 1"

[1333] In one embodiment of the present invention, a worker wears a head-mounted display. This display has a built-in camera and microphone, and an emotion engine. The worker starts a task, takes an image of the part with the camera of the head-mounted display, and records questions about the task with the microphone. These images and voices are input to the generation AI. The generation AI analyzes the images and voices and determines the next task procedure. Furthermore, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next task procedure based on the emotions. For example, if the emotion engine senses that the worker is stressed, it outputs an instruction to encourage the worker to take a break.

[1334] "Example 2"

[1335] In another embodiment of the present invention, a worker starts assembling an automobile engine, takes images of engine parts with a camera on a head-mounted display, and records a question with a microphone: "Which part should I install next?" These images and voices are input to a generative AI. The generative AI analyzes the images and voices and determines the next part to be installed and the order in which to install it. Furthermore, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on the emotions. For example, if the worker feels tired, the emotion engine outputs an instruction to encourage the worker to take a break.

[1336] The process flow of each embodiment will be described below.

[1337] "Example 1"

[1338] Step 1: The worker wears a head-mounted display, which has a built-in camera, microphone, and emotion engine.

[1339] Step 2: The worker begins the task, taking images of the part with the camera in the head-mounted display and recording questions about the task with the microphone.

[1340] Step 3: These images and audio are input into the generative AI.

[1341] Step 4: The generative AI analyzes the video and audio and determines the next steps.

[1342] Step 5: The emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on those emotions.

[1343] Step 6: For example, if the worker feels stressed, the emotion engine outputs an instruction to encourage the worker to take a break.

[1344] "Example 2"

[1345] Step 1: A worker begins assembling a car engine, taking images of the engine parts with a camera in a head-mounted display and recording a question with a microphone: "Which part should I install next?"

[1346] Step 2: These images and audio are input into the generative AI.

[1347] Step 3: The generative AI analyzes the video and audio to determine the next part to be installed and the order in which it should be installed.

[1348] Step 4: The emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on those emotions.

[1349] Step 5: For example, if a worker feels tired, the emotion engine outputs an instruction to encourage the worker to take a break.

[1350] Example 1

[1351] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1352] In conventional work support systems, it is sometimes difficult to give appropriate instructions on the next work procedure that a worker should perform. In addition, because the system gives uniform instructions without considering the worker's emotional state, it is difficult to reduce the worker's stress and fatigue. This reduces work efficiency and increases the possibility of work errors.

[1353] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1354] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the voice of the worker and adjust the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate instructions for work procedures that take into account the emotional state of the worker, which is expected to improve work efficiency and reduce work errors.

[1355] "Worker" refers to a person who performs a particular task.

[1356] "Camera" refers to a device for capturing images.

[1357] "Microphone" refers to a device for recording sound.

[1358] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[1359] "Input" refers to entering data into a system.

[1360] "Output" refers to sending data out of the system.

[1361] An "emotion engine" is a system that recognizes emotions from voice and video, and generates appropriate instructions based on that information.

[1362] "Video" refers to visual information captured by a camera.

[1363] "Audio" refers to auditory information recorded by a microphone.

[1364] "Work procedure" refers to the specific steps or methods for performing a particular task.

[1365] "Instructions" refer to information intended to encourage a particular action.

[1366] A "system" refers to a series of devices or programs in which multiple elements work together.

[1367] This invention is a system in which a worker wears a head-mounted display and inputs video and audio to a generative AI via the display's built-in camera and microphone. The generative AI analyzes the video of the parts the worker should handle and the worker's audio (e.g., "Which part should I install next?") and determines the next work procedure. The instructions are communicated to the worker via the display and speaker of the head-mounted display.

[1368] Hardware and Software Used

[1369] Hardware: Head-mounted display (built-in camera, microphone, display, speaker)

[1370] Software: Generative AI models, emotion engines

[1371] Specific operation of the system

[1372] Subject: Server

[1373] The server receives video and audio data transmitted from the head-mounted display worn by the worker. This data is collected through the built-in camera and microphone. The server analyzes the received video data and identifies the part that the worker should handle. It also analyzes the audio data to understand the worker's question (e.g., "Which part should I install next?"). Based on the results of this analysis, the server determines the next work procedure and conveys those instructions to the worker via the display and speakers of the head-mounted display.

[1374] Subject: Terminal

[1375] The terminal (head-mounted display) is a device worn by the worker. The terminal uses a built-in camera and microphone to collect images of parts within the worker's field of vision and the worker's voice. The collected data is sent to the server in real time. When the terminal receives instructions from the server, it displays the instructions on the display and conveys them to the worker by voice through the speaker.

[1376] Subject: User

[1377] The user (worker) wears the head-mounted display while performing their work. As the user handles parts, the built-in camera captures the image, and the microphone records any questions about the work. When the user asks, "Which part should I install next?", the voice data is sent to the server. After receiving instructions from the server, the user checks the next work procedure through the display and speaker of the head-mounted display.

[1378] Examples

[1379] Example 1: Parts installation

[1380] When a user asks, "Which part should I attach next?", the microphone in the head-mounted display records the voice and sends it to the server. The server analyzes the video and audio data to identify the next part to be attached. For example, if the server instructs, "Attach the screw next," the instruction is displayed on the display of the head-mounted display and communicated by voice through the speaker. The user follows the instruction to attach the screw.

[1381] Example 2: Emotion-recognition-based break instruction

[1382] If a user says "I'm tired" while working, the microphone in the head-mounted display records the voice and sends it to the server. The emotion engine recognizes the user's stress from the voice. The emotion engine generates an instruction to encourage the user to take a break, and the server sends the instruction to the head-mounted display. The device displays the instruction "Let's take a short break" on the display and conveys it by voice through the speaker. The user follows the instruction and takes a break.

[1383] Examples of prompt statements

[1384] "If a worker wears a head-mounted display and asks what the next step is, please explain how the generative AI model would determine the next step and communicate it to the worker."

[1385] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1386] Step 1:

[1387] The user wears a head-mounted display.

[1388] Specifically, the user wears the head-mounted display and checks that the built-in camera and microphone are working properly, enabling the collection of video and audio.

[1389] Step 2:

[1390] The device collects video and audio.

[1391] Specifically, the device's built-in camera captures images of parts that are within the user's field of vision, and the built-in microphone records the user's voice. For example, if the user asks, "Which part should I install next?", that voice is recorded.

[1392] Input: Images of parts in the user's field of vision, user's voice

[1393] Output: Collected video and audio data

[1394] Step 3:

[1395] The terminal transmits the collected data to the server.

[1396] Specifically, the device transmits collected video and audio data in real time to the server, which then prepares the data for analysis.

[1397] Input: Collected video and audio data

[1398] Output: Data sent to the server

[1399] Step 4:

[1400] The server parses the data.

[1401] Specifically, the server analyzes the received video data to identify the part the user is handling. It also analyzes the voice data to understand the user's question. For example, if the user asks, "Which part should I install next?", the server analyzes and understands the question.

[1402] Input: Video and audio data sent to the server

[1403] Output: Analysis results (part identification, question understanding)

[1404] Step 5:

[1405] The server decides the next step to take.

[1406] Specifically, the server determines the next step based on the analysis results. For example, if the user determines that the part to be installed is a screw, the server generates the instruction "Install the screw next."

[1407] Input: Analysis results (part identification, understanding of question content)

[1408] Output: Next step instructions

[1409] Step 6:

[1410] The server sends the instruction to the terminal.

[1411] Specifically, the server sends instructions for the determined work procedure to the terminal, including the specific work that the user should do next.

[1412] Input: Next step instructions

[1413] Output: Instructions sent to the terminal

[1414] Step 7:

[1415] The terminal conveys the instructions to the user.

[1416] Specifically, the terminal conveys the instructions received from the server to the user. Specifically, the instructions are displayed on the head-mounted display and conveyed by voice through the speaker. For example, the instruction "Next, install the screws" is displayed on the display and conveyed by voice.

[1417] Input: Instructions sent to the terminal

[1418] Output: Instructions given to the user

[1419] (Application example 1)

[1420] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1421] In conventional factory work, workers had to refer to the manual or ask their superiors to confirm the next work procedure, which led to problems of reduced work efficiency. In addition, work instructions were given without considering the emotional state of the workers, which increased the workers' stress and increased the risk of mistakes and accidents.

[1422] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1423] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the voice of the worker and adjust the instructions for the next work procedure based on the emotions. This allows the worker to not only check the next work procedure in real time, but also to receive appropriate instructions according to his / her emotional state.

[1424] "Worker" refers to the person who actually performs the work in a factory or on-site.

[1425] "Camera" refers to a device that captures images and records them as digital data.

[1426] A "microphone" refers to a device that records sound and saves it as digital data.

[1427] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[1428] An "emotion engine" refers to a system that recognizes emotions from voice and video, and generates appropriate instructions based on the results.

[1429] "Video" refers to visual information captured through a camera.

[1430] "Speech" refers to auditory information captured through a microphone.

[1431] "Means" refers to a method or device for achieving a specific purpose.

[1432] A "system" refers to a set of devices or programs in which multiple elements function in conjunction with one another.

[1433] A "work procedure" refers to the series of steps or instructions required to complete a particular task.

[1434] "Output" refers to the output of results generated by the system to the outside.

[1435] "Input" refers to entering data into a system.

[1436] The system for implementing this invention captures video and audio using a camera and microphone held by the worker, and inputs this to the generation AI. The generation AI analyzes this data and determines the next work procedure. The emotion engine also recognizes emotions from the worker's voice and adjusts instructions based on those emotions.

[1437] Hardware and Software Used

[1438] Hardware: Head-mounted display (HMD), built-in camera, built-in microphone

[1439] Software: OpenCV (camera image acquisition), SpeechRecognition (voice acquisition), EmotionRecognizer (emotion recognition), AIDecisionMaker (generative AI model)

[1440] System Operation

[1441] The server acquires video through the HMD's camera and audio through the microphone. This data is input to the generative AI in real time. The generative AI analyzes the video and audio and determines the next work procedure. For example, if a worker asks, "Which part should I install next?" the generative AI analyzes the video and audio and issues the instruction, "Install the screws next."

[1442] Furthermore, the emotion engine recognizes the emotion of the worker from the voice. For example, if the worker is feeling stressed, the emotion engine will instruct the worker to "take a break." In this way, appropriate instructions are provided according to the worker's emotional state.

[1443] Examples

[1444] Take factory assembly work as an example. A worker wears an HMD, takes pictures of parts with the built-in camera, and records questions about the work with the built-in microphone. This data is input into the generative AI, which determines the next work procedure. For example, if a worker asks, "Which part should I attach next?" the generative AI will instruct, "Attach the screws next." If the worker is feeling stressed, the emotion engine will instruct, "Take a break."

[1445] Examples of prompt statements

[1446] An example of a prompt sentence to be input to the generative AI model is as follows.

[1447] The worker is asking, "Which part should I install next?" The video shows parts A and B. Please tell me the next step.

[1448] By inputting this prompt into a generative AI model, the next steps can be determined.

[1449] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1450] Step 1:

[1451] The server acquires images through the HMD camera. The image data from the camera is captured in real time and stored as digital data. This image data includes visual information of the parts handled by the worker and the work environment.

[1452] Step 2:

[1453] The server receives voice from the microphone of the HMD. The voice data from the microphone is captured in real time and stored as digital data. This voice data includes the worker's questions and instructions.

[1454] Step 3:

[1455] The server inputs the acquired video and audio data into the generation AI, which then analyzes the data and determines the next work procedure. Specifically, it identifies the type and location of parts from the video data, and understands the content of the worker's questions from the audio data.

[1456] Step 4:

[1457] The generative AI determines the next work procedure based on the analysis results and generates the instructions. For example, if a worker asks, "Which part should I attach next?", the generative AI generates the instruction, "Attach the screws next."

[1458] Step 5:

[1459] The server outputs instructions from the generation AI to the worker through the HMD's display and speakers. Visual instructions are displayed on the display, and audio instructions are played back from the speakers.

[1460] Step 6:

[1461] The server uses an emotion engine to recognize the worker's emotions from the voice data. The emotion engine analyzes the tone, speed, and content of the voice to determine whether the worker is feeling stressed.

[1462] Step 7:

[1463] The emotion engine recognizes the worker's emotions and adjusts the instructions of the generative AI based on the results. For example, if a worker is feeling stressed, the emotion engine will generate the instruction "Please take a break."

[1464] Step 8:

[1465] The server outputs the adjusted instructions to the worker again through the HMD display and speakers, allowing the worker to receive appropriate instructions according to their emotional state.

[1466] Example 2

[1467] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1468] Conventional work support systems have difficulty instructing workers on the next steps, and are particularly unable to provide instructions that take into account the worker's emotions and fatigue state. This leads to problems such as reduced work efficiency and increased worker fatigue and stress.

[1469] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1470] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for recognizing emotions from the voice of the worker using an emotion engine and adjusting the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate instructions for work procedures that take into account the emotions and fatigue state of the worker.

[1471] "Worker" refers to a person who performs work such as assembling or repairing machines, vehicles, etc.

[1472] "Camera" refers to a device for capturing images.

[1473] "Microphone" refers to a device for recording sound.

[1474] "Generative AI" refers to an artificial intelligence model that analyzes input data and generates an appropriate output.

[1475] An "emotion engine" refers to technology for recognizing emotions from data such as audio and video.

[1476] "Video" refers to visual information captured through a camera.

[1477] "Speech" refers to auditory information captured through a microphone.

[1478] "Input" refers to entering data into a system.

[1479] "Output" refers to sending data out of the system.

[1480] "Means" refers to a method or device for achieving a specific purpose.

[1481] "Work procedure" refers to the specific steps and methods for proceeding with work.

[1482] "Instructions" refers to communicating to workers the actions and procedures they should follow.

[1483] "Image" refers to a still image or video used to express visual information.

[1484] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1485] This invention is a system that uses a head-mounted display (HMD) to instruct workers on the next part to be installed and the procedure for installing it when assembling or repairing machines, vehicles, etc. The system is composed of a generative AI model that performs video and audio analysis, and an emotion engine that recognizes emotions.

[1486] Hardware and Software Used

[1487] Head-mounted display (HMD): Equipped with a camera and microphone, it captures the worker's visual and auditory information.

[1488] Server: Provides the computational resources to run the generative AI model and the emotion engine.

[1489] Generative AI model: Analyzes video and audio data to determine which part to install next and in what order.

[1490] Emotion engine: Recognizes emotions from a worker's voice and adjusts instructions for the next work step based on those emotions.

[1491] System Operation

[1492] 1. The user puts on the HMD:

[1493] The user puts on the HMD and starts assembling engine parts. The HMD has a built-in camera and microphone, which are used to record the progress of the work.

[1494] 2. The device acquires video and audio:

[1495] The device (HMD) uses a camera to capture images of engine parts in real time and a microphone to record the user's voice, for example when the user asks, "Which part should I install next?"

[1496] 3. The device sends the data to the server:

[1497] The device transmits captured video and recorded audio data to a server, where it is used for analysis by generative AI models and emotion engines.

[1498] 4. The server analyzes the data:

[1499] The server inputs the received video and audio data into a generative AI model. The generative AI model analyzes the state of the engine parts from the video and understands the user's question from the audio. For example, it recognizes from the video that the piston has not yet been installed, and understands from the audio that the user is asking about the next part to be installed.

[1500] 5. The server recognizes emotions:

[1501] The server uses an emotion engine to recognize emotions from the user's voice, for example if the user's voice sounds tired.

[1502] 6. The server generates the instruction:

[1503] The server determines the next part to be installed and the procedure based on the analysis results of the generative AI model and the recognition results of the emotion engine. For example, it generates an instruction such as "Install the piston next." If it recognizes that the user is tired, it also adds an instruction such as "Take a short break."

[1504] 7. The terminal will output the command:

[1505] The terminal receives instructions from the server, displays them on the HMD display as an image showing the position of the parts, and outputs voice instructions from the speaker such as "Next, attach the piston." If the user is tired, it also outputs voice instructions such as "Please take a short break."

[1506] Examples

[1507] Let's say a user is assembling an engine. The HMD camera captures video of the engine parts and asks, "Which part should I install next?" through the microphone. The device sends this data to the server. The server uses a generative AI model to analyze the video and audio, and generates the instruction, "Install the piston next." Furthermore, if the emotion engine recognizes fatigue from the user's voice, it also adds the instruction, "Please take a short break." The device communicates these instructions to the user through the HMD's display and speaker.

[1508] Examples of prompt statements

[1509] "We are currently assembling the engine. What is the next part to be installed and in what order? Please analyze the current video and audio data."

[1510] This system combines an HMD, a generative AI model, and an emotion engine to improve the user's work efficiency and provide flexible instructions according to the worker's condition.

[1511] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1512] Step 1:

[1513] The user puts on the HMD and starts assembling engine parts. The user uses the camera and microphone of the HMD to record the progress of the work. Specifically, the user asks, "Which part should I install next?"

[1514] Input: User's audio and video

[1515] Output: Audio and video data recorded in the HMD

[1516] Step 2:

[1517] The device (HMD) captures images of engine parts and records audio. The camera captures images in real time, and the microphone records the user's questions. For example, the device captures images of engine parts and audio such as "Which part should I install next?"

[1518] Input: User's audio and video

[1519] Output: Shooting video data and recording audio data

[1520] Step 3:

[1521] The device transmits video and audio data to the server. The HMD transmits the captured data to the server, where it is used for analysis by the generative AI model and emotion engine.

[1522] Input: Recorded video data and recorded audio data

[1523] Output: Video and audio data sent to the server

[1524] Step 4:

[1525] The server analyzes the video and audio using a generative AI model. The server inputs the received video and audio data into the generative AI model, analyzes the state of the engine parts from the video, and understands the user's question from the audio. For example, it recognizes from the video that the piston has not yet been installed, and understands from the audio that the user is asking about the next part to be installed.

[1526] Input: Video and audio data sent to the server

[1527] Output: Analyzed part status and user question

[1528] Step 5:

[1529] The server uses the emotion engine to recognize the user's emotion. The server uses the emotion engine to recognize the emotion from the user's voice. For example, if the user's voice sounds tired, the server recognizes that emotion.

[1530] Input: User's voice data

[1531] Output: The perceived emotional state of the user.

[1532] Step 6:

[1533] The server generates instructions based on the analysis results and emotion recognition results. Based on the analysis results of the generative AI model and the recognition results of the emotion engine, the server determines the next part to be installed and the procedure for doing so. For example, it generates an instruction such as "Install the piston next." If it recognizes that the user is tired, it also adds an instruction such as "Take a short break."

[1534] Input: Analyzed part states and the user's emotional state

[1535] Output: Generated work instructions

[1536] Step 7:

[1537] The terminal displays and speaks instructions to the user. The terminal receives instructions from the server and displays them on the HMD display as an image showing the positions of parts. It also outputs voice instructions from the speaker, such as "Next, attach the piston." If the user is tired, it also outputs voice instructions such as "Please take a short break."

[1538] Input: Generated work order

[1539] Output: Images displayed on the HMD display and audio instructions output from the speaker.

[1540] (Application example 2)

[1541] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1542] Conventional assembly work support systems have difficulty instructing workers on the next work procedure, which reduces work efficiency. In addition, they are unable to give instructions that take into account the worker's fatigue and stress, which can have a negative impact on the worker's health and work quality. To solve these problems, a system that recognizes the worker's emotions and issues appropriate instructions based on them is needed.

[1543] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1544] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the worker's voice and adjust the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate work instructions that take into account the emotions of the worker, improving work efficiency and maintaining the health of the worker.

[1545] A "camera" is a device for capturing images.

[1546] A "microphone" is a device for capturing sound.

[1547] "Generative AI" is artificial intelligence that analyzes input data and generates appropriate output.

[1548] An "emotion engine" is a system that recognizes emotions from data such as audio and video, and generates appropriate instructions based on those emotions.

[1549] A "task procedure" is the sequence of steps required to complete a particular task.

[1550] "Voice" refers to words and sounds made by the worker.

[1551] "Video" is visual information captured through a camera.

[1552] "Output" refers to the provision of instructions and information generated by the system to workers.

[1553] "Input" refers to data or information entered into a system.

[1554] A "system" is a configuration in which multiple devices and software work together to achieve a specific function.

[1555] The system for implementing this invention captures video and audio using a camera and microphone held by the worker, and inputs them into the generation AI. The generation AI analyzes the next work procedure and outputs it to the worker as audio or images. The emotion engine also recognizes emotions from the worker's voice and adjusts the instructions for the next work procedure based on those emotions.

[1556] Hardware and Software Used

[1557] Hardware:

[1558] Camera (e.g. Logitech C920)

[1559] Microphone (e.g. Blue Yeti)

[1560] Head-mounted displays (e.g. Microsoft HoloLens)

[1561] software:

[1562] OpenCV (for image capture)

[1563] SpeechRecognition (for voice capture)

[1564] Transformers (using generative AI and sentiment analysis models)

[1565] Data processing and calculation

[1566] The server receives the video captured through the camera and the audio captured through the microphone. These data are input to the generative AI model, which generates the next steps to be taken. The generated instructions are displayed on the head-mounted display or communicated to the worker by voice. In addition, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions based on those emotions.

[1567] Examples

[1568] When a worker asks, "Which part should I install next?", a camera takes a picture of the engine part and inputs it into the generative AI model. The generative AI model generates an instruction, such as "Install the piston next," and displays it on the head-mounted display. Furthermore, if the emotion engine senses fatigue from the worker's voice, it issues the instruction, "Please take a break."

[1569] Examples of prompt statements

[1570] Image: <Engine parts image>

[1571] Q: What part should I install next?

[1572] Please tell me what parts I should install next and what steps to take.

[1573] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1574] Step 1:

[1575] A user uses a camera to capture video of an engine part.

[1576] Input: Physical condition of engine parts

[1577] Output: Captured image data of engine parts

[1578] Specific operation: A camera (e.g. Logitech C920) is used to capture images of engine parts. The image data is sent to the server in real time.

[1579] Step 2:

[1580] The user uses a microphone to capture audio saying, "Which part should I install next?"

[1581] Input: User's voice

[1582] Output: Captured audio data

[1583] What it does: Use a microphone (e.g. Blue Yeti) to record the user's question. The audio data is sent to the server in real time.

[1584] Step 3:

[1585] The server inputs the captured video and audio into a generative AI model.

[1586] Input: Video data, audio data

[1587] Output: Input data for the generative AI model

[1588] How it works: The server receives the captured video and audio and formats the data for input into a generative AI model (e.g. GPT-3.5-turbo).

[1589] Step 4:

[1590] The generative AI model analyzes the next work step and generates instructions.

[1591] Input: Input data (video data, audio data)

[1592] Output: Work procedure instructions

[1593] Specific operation: The generative AI model analyzes the input data and determines which parts to install next and in what order, for example generating instructions such as "Install the piston next."

[1594] Step 5:

[1595] The server displays the generated instructions on a head-mounted display or conveys them to the user by voice.

[1596] Input: Work procedure instructions

[1597] Output: Display or audio instructions

[1598] Specific operation: The server displays the generated instructions on the display of a head-mounted display (e.g., Microsoft HoloLens) or conveys them to the user audibly through the speakers.

[1599] Step 6:

[1600] The server uses an emotion engine to recognize emotions from the user's voice.

[1601] Input: User's voice data

[1602] Output: Emotion recognition result

[1603] Specific operation: The emotion engine analyzes the user's voice data and recognizes the user's emotions (e.g., tiredness, stress).

[1604] Step 7:

[1605] The server adjusts the instructions for the next work step based on the emotion.

[1606] Input: Emotion recognition results

[1607] Output: Coordinated work instructions

[1608] Specific Actions: Based on the results of the emotion engine, the server adjusts the next steps of the work. For example, if the server detects that the user is tired, it will give the instruction "Please take a break."

[1609] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1610] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1611] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1612] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1613] [Fourth embodiment]

[1614] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1615] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1616] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1617] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1618] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1619] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1620] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1621] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1622] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1623] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1624] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1625] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1626] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1627] "Example 1"

[1628] In one embodiment of the present invention, a worker wears a head-mounted display and inputs video and audio to the generation AI through the display's built-in camera and microphone. The generation AI analyzes the video of the parts the worker should handle and the worker's audio (e.g., "Which part should I install next?") and determines the next work procedure. The instructions are conveyed to the worker through the display and speaker of the head-mounted display.

[1629] "Example 2"

[1630] As a specific example, consider the assembly of an automobile engine. A worker wears a head-mounted display and inputs images of engine parts and the voice saying, "Which part should I install next?" into the generative AI. The generative AI analyzes the video and voice and determines the next part to be installed and the order in which to install it. The instructions are communicated to the worker as an image showing the position of the parts on the head-mounted display screen and as a voice message over the speaker saying, "Install the piston next."

[1631] The process flow of each embodiment will be described below.

[1632] "Example 1"

[1633] Step 1: The worker wears a head-mounted display, which has a built-in camera and microphone.

[1634] Step 2: The worker starts the task, takes a picture of the part with the camera in the head-mounted display, and records a question about the task with the microphone. These images and voices are input to the generative AI.

[1635] Step 3: The generative AI analyzes the video and audio and determines the next steps to take.

[1636] Step 4: The generating AI communicates instructions for the next work procedure to the worker via the head-mounted display and speakers.

[1637] "Example 2"

[1638] Step 1: A worker starts assembling a car engine, takes pictures of the engine parts with a camera on a head-mounted display, and uses a microphone to record a question such as, "Which part should I install next?" These images and voices are input to the generative AI.

[1639] Step 2: The generative AI analyzes the video and audio to determine the next part to be installed and the order in which it should be installed.

[1640] Step 3: The generating AI communicates this to the worker in the form of an image on the head-mounted display showing the position of the parts, and a voice message over the speaker saying, "Next, install the piston."

[1641] Example 1

[1642] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1643] Conventional work assistance systems have had difficulty efficiently understanding the next work procedure that a worker should perform. In addition, they were unable to analyze data collected through the camera and microphone carried by the worker in real time and provide appropriate instructions. This resulted in issues such as reduced work efficiency and an increased risk of mistakes.

[1644] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1645] In this invention, the server includes a means for inputting video and audio through a display device worn by a worker to the generative AI model, a means for the generative AI model to output instructions for the next work procedure to the worker by audio or image, a means for the display device to transmit collected data to the server, a means for the server to transmit the results of the generative AI model to the display device, and a means for the display device to present the work procedure to the worker. This allows the worker to grasp the next work procedure in real time, improving work efficiency and reducing the occurrence of errors.

[1646] A "worker" is a person who wears a display device and uses the system to perform a specific task.

[1647] A "display device" is a device such as a head-mounted display worn by a worker, which has the function of collecting video and audio and displaying instructions.

[1648] A "generative AI model" is an artificial intelligence model that analyzes collected video and audio data and determines the next work steps.

[1649] The "server" is a computer system whose role is to receive data sent from the display device, input it into the generative AI model, and send the results to the display device.

[1650] "Image" refers to visual information of the work environment and parts captured through the display device's camera.

[1651] "Voice" refers to acoustic information of the worker's comments and questions collected through the microphone of the display device.

[1652] "Input means" refers to a method or device for inputting the video and audio collected by the display device into the generative AI model.

[1653] "Means for output" refers to a method or device for providing the work procedures determined by the generative AI model to workers in the form of audio or images.

[1654] "Transmitting means" refers to a method or apparatus for the display device to transmit collected data to the server and for the server to transmit the results of the generative AI model to the display device.

[1655] "Presenting means" means a method or device by which a display device presents the results of the generative AI model to a human operator visually or audibly.

[1656] This invention is a system that collects video and audio through a display device worn by a worker and inputs them into a generative AI model to provide the next work procedure in real time. A specific embodiment of this system will be described below.

[1657] First, the user wears a head-mounted display (HMD). This HMD has a built-in camera and microphone, and has the function of collecting video and audio of the work environment. The HMD can be a commercially available device such as Microsoft HoloLens or Oculus Rift.

[1658] The device (HMD) uses a camera to capture images of the parts and work environment within the user's field of vision, and records the user's comments and questions with a microphone. This data is sent to a server in real time. Data is sent using wireless communication technologies such as Wi-Fi and Bluetooth.

[1659] The server inputs the received video and audio data into a generative AI model, which is built using frameworks such as TensorFlow and PyTorch. The generative AI model analyzes the video data to identify the part the user is handling, and analyzes the audio data to understand the user's question.

[1660] Once the generative AI model has determined the next work procedure, the result is sent from the server to the device. The device then presents the received work procedure to the user. Specifically, the HMD display shows an image and explanation of the next part to be installed, and the speaker issues audio instructions.

[1661] As a concrete example, consider the case where a user is assembling a car engine. The user is wearing an HMD and handling engine parts. When the user asks, "Which part should I install next?", the HMD's camera captures the current work situation and the microphone records the question. This data is sent to the server, where it is analyzed by the generative AI model. The generative AI model determines that the next part to be installed is a "piston" and returns this information to the server. The server sends the result to the terminal, which displays "Install the piston next" on the HMD's display and also plays the same instruction over the speaker.

[1662] Example prompt:

[1663] A worker is assembling an engine. Capture the current work status with a camera, analyze the worker's question "Which part should I install next?", and instruct the worker on which part to install next.

[1664] In this way, the system allows the server, terminals, and users to work together to efficiently carry out tasks.

[1665] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1666] Step 1:

[1667] The user puts on the HMD and begins working.

[1668] Specifically, the user puts on the HMD and turns it on. This activates the HMD's camera and microphone, allowing video and audio to be collected. The input is the video and audio of the parts and work environment within the user's field of vision, and the output is the data collected through the HMD's camera and microphone.

[1669] Step 2:

[1670] The terminal (HMD) collects video and audio.

[1671] Specifically, the HMD camera captures images of the parts and working environment within the user's field of vision, and the microphone records the user's comments and questions. The input is the images and audio of the parts and working environment within the user's field of vision, and the output is the collected video and audio data.

[1672] Step 3:

[1673] The terminal transmits the collected data to the server.

[1674] Specifically, the HMD compresses the collected video and audio data and transmits it to the server using wireless communication technologies such as Wi-Fi and Bluetooth. The input is the collected video and audio data, and the output is the data transmitted to the server.

[1675] Step 4:

[1676] The server inputs the data into a generative AI model.

[1677] Specifically, the server inputs the received video and audio data into a generative AI model. The generative AI model is built using frameworks such as TensorFlow and PyTorch. The input is the video and audio data sent to the server, and the output is the data input into the generative AI model.

[1678] Step 5:

[1679] A generative AI model analyzes the data and determines the next steps to take.

[1680] Specifically, the generative AI model analyzes video data to identify the part the user is handling, and analyzes audio data to understand the user's question. For example, if a user asks, "Which part should I install next?", the generative AI model analyzes the question and determines which part should be installed next. The input is the video and audio data input to the generative AI model, and the output is instructions for the next work procedure.

[1681] Step 6:

[1682] The server sends the results of the generated AI model to the device.

[1683] Specifically, the server sends the next work procedure determined by the generative AI model to the terminal. The data is compressed again and sent to the terminal via the network. The input is the result of the generative AI model, and the output is the work procedure instructions sent to the terminal.

[1684] Step 7:

[1685] The terminal presents the work procedure to the user.

[1686] Specifically, the terminal presents the received work procedure to the user. An image and explanation of the next part to be installed is shown on the HMD display, and instructions are given by voice from the speaker. For example, instructions such as "Install the piston next" are displayed. The input is the work procedure instructions sent to the terminal, and the output is the work procedure presented to the user.

[1687] (Application example 1)

[1688] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1689] In conventional assembly and repair work in factories, workers have to refer to the manual to confirm the next step, which reduces work efficiency. There is also a risk that the quality of work will decline if the worker makes a mistake in the next step. To solve these problems, a system that can provide instructions on work procedures in real time is needed.

[1690] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1691] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, a means for the generation AI to analyze the voice command and image of the worker and determine the next work procedure, and a means for communicating the determined work procedure to the worker through the display and speaker of the head mounted display. This allows the worker to check the next work procedure in real time, making it possible to improve work efficiency and maintain work quality.

[1692] "Worker" refers to a person who assembles or repairs machines or parts in a factory or work site.

[1693] "Camera" refers to a device used to capture video and input the data into generative AI.

[1694] A "microphone" refers to a device for recording audio and inputting the data into the generative AI.

[1695] "Generative AI" refers to artificial intelligence that analyzes input video and audio and determines the next steps in the process.

[1696] A "head-mounted display" refers to a device worn by a worker to display and play back images and audio.

[1697] "Display" refers to a screen for visually displaying the work procedures determined by the generation AI.

[1698] "Speaker" refers to a device for playing back the work procedures determined by the generation AI in audio form.

[1699] "Voice command" refers to a voice instruction given by a worker to inquire about the next work procedure.

[1700] "Video" refers to visual information captured through a camera.

[1701] "Work procedure" refers to the specific operation or process that a worker should perform next.

[1702] The system for implementing this invention uses a camera and microphone held by a worker to input video and audio to a generation AI, which then determines the next work procedure and issues instructions to the worker. This system uses the following hardware and software.

[1703] Hardware

[1704] 1. Camera: The camera carried by the worker is used to capture footage of the work site and input the data into the generative AI.

[1705] 2. Microphone: The microphone held by the worker is used to record the worker's voice and input the data into the generative AI.

[1706] 3. Head-Mounted Display (HMD): The HMD worn by the worker is equipped with a display that visually displays the work procedures determined by the generation AI, and speakers that play them audibly.

[1707] software

[1708] 1. OpenCV: Used to acquire and process camera images.

[1709] 2. SpeechRecognition: Used to convert audio recorded from a microphone into text.

[1710] 3. Transformers: Uses generative AI models (e.g. GPT-3) to analyze input video and audio and determine the next steps.

[1711] System Operation

[1712] 1. Acquiring video and audio: The video of the work site and the voice of the worker are acquired through the camera and microphone held by the worker. The camera acquires video data using OpenCV, and the microphone converts the voice data into text using SpeechRecognition.

[1713] 2. Input to the generative AI: The captured video and audio data are input into the generative AI model, which uses Transformers to analyze the video and audio commands and determine the next steps.

[1714] 3. Output of work procedures: The work procedures determined by the generation AI are visually displayed on the HMD display and played audibly through the speaker.

[1715] Examples

[1716] For example, if a worker on a factory assembly line asks, "Which part should I install next?", a camera captures video of the work site and a microphone records the audio. The generative AI model uses the following prompt to determine the next step:

[1717] Example prompt:

[1718] Image: [Part image data], Voice command: "Which part should I install next?" What's the next step?

[1719] The generative AI model analyzes the prompt and determines specific work steps, such as "Next, install the screw." These instructions are displayed on the HMD display and communicated to the worker through the speaker.

[1720] In this way, workers can check the next work procedure in real time, improving work efficiency and maintaining work quality.

[1721] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1722] Step 1:

[1723] The server obtains images from the camera held by the worker. The camera captures images of the work site in real time and processes the data using OpenCV. The input is the image data from the camera, and the output is the processed image data.

[1724] Step 2:

[1725] The server captures the voice from the worker's microphone. The microphone records the worker's voice in real time and converts the data into text using SpeechRecognition. The input is the voice data from the microphone, and the output is the converted text data.

[1726] Step 3:

[1727] The server inputs the acquired video data and text data into the generative AI model. The generative AI model uses Transformers to analyze the video and voice commands and determine the next work steps. The input is video data and text data, and the output is text data regarding the next work steps.

[1728] Step 4:

[1729] The server displays the next work procedure obtained from the generative AI model on the display of a head-mounted display (HMD). The display visually shows the specific operation or process that the worker should perform next. The input is text data regarding the next work procedure, and the output is visual information displayed on the display.

[1730] Step 5:

[1731] The server plays the next work procedure obtained from the generative AI model aloud through the speaker of the head-mounted display (HMD). The speaker aloud tells the worker the specific operation or process that he or she should perform next. The input is text data regarding the next work procedure, and the output is audio information played from the speaker.

[1732] Step 6:

[1733] The user confirms the next steps through the HMD's display and speakers, and performs the work according to the instructions. The input is visual and audio information from the display and speakers, and the output is the work that the worker actually performs.

[1734] Example 2

[1735] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1736] With conventional work support systems, it was difficult to efficiently instruct workers on the next step in the work process. In addition, workers had to stop working to confirm the next step, which reduced work efficiency. Furthermore, it was difficult for workers to accurately grasp the location of parts, which meant there was a risk of installing the wrong parts.

[1737] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1738] In this invention, the server includes a means for attaching a display device held by a worker, a means for acquiring an image through a camera and a voice of the worker through a microphone, a means for analyzing the acquired image with image analysis software and identifying the type and position of parts, a means for converting the acquired voice into text data with voice recognition software, a means for inputting the image analysis result and the voice recognition result into a generative AI model, a means for the generative AI model to determine the next part to be installed and its procedure, a means for outputting the instructions determined by the generative AI model as text data, a means for displaying an image showing the position of the parts on the display of the display device, and a means for generating instructions by voice using voice synthesis software and communicating them to the worker through a speaker. This allows the worker to check the next work procedure without stopping his / her work and to accurately grasp the position of the parts.

[1739] A "display device" is a device worn by a worker to provide visual information.

[1740] A "camera" is a device for taking images and acquiring the data.

[1741] A "microphone" is a device for recording sound and acquiring the data.

[1742] "Image analysis software" is software for analyzing acquired video data and extracting specific information.

[1743] "Voice recognition software" refers to software for converting acquired voice data into text data.

[1744] A "generative AI model" is an artificial intelligence model that analyzes input data and determines the next work procedure to be performed.

[1745] "Text data" is character information converted by voice recognition software.

[1746] A "display" is a screen built into a display device for displaying visual information.

[1747] "Speech synthesis software" means software for converting text data into speech and generating spoken instructions.

[1748] A "speaker" is a device for outputting speech generated by speech synthesis software.

[1749] This invention is a system in which a worker wears a display device, analyzes video and audio captured through a camera and microphone, and instructs the next work procedure using a generative AI model. A specific embodiment of this system will be described below.

[1750] Hardware and Software Used

[1751] Hardware: Display device (head-mounted display), camera, microphone, speaker

[1752] Software: Image analysis software, voice recognition software, generative AI models, voice synthesis software

[1753] System Operation

[1754] 1. The user wears a display device (head-mounted display) that has a built-in camera and microphone.

[1755] 2. The user takes a video of the engine parts with a camera and inputs into the microphone, "Which part should I install next?"

[1756] 3. The device captures the video captured by the camera and the audio recorded by the microphone.

[1757] 4. The video captured by the device is analyzed using image analysis software to identify the type and location of the engine parts.

[1758] 5. The voice captured by the device is converted into text data using voice recognition software.

[1759] 6. The device inputs the image analysis results and voice recognition results into the generative AI model.

[1760] 7. The generative AI model determines which part to install next and in what order.

[1761] 8. The instructions determined by the generative AI model are output as text data.

[1762] 9. The terminal receives instructions from the generative AI model and displays an image showing the location of the parts on the display device.

[1763] 10. The device uses speech synthesis software to generate a voice message saying, "Next, install the piston," and communicates this to the user through the speaker.

[1764] Examples

[1765] For example, say a user is assembling an engine. The user puts on the display device, takes pictures of the engine parts with the camera, and speaks into the microphone saying, "Which part should I install next?" The device captures the video and audio, uses image analysis software to identify the type and location of the part, and uses voice recognition software to convert the speech to text. This data is then input into a generative AI model that determines which part should be installed next and in what order. For example, an instruction might be generated: "Install the piston next." The device receives this instruction, displays an image showing the position of the piston on the display device's screen, and transmits the instruction via voice through the speaker.

[1766] Examples of prompt statements

[1767] "Analyze video and audio of engine parts to determine which part should be installed next and in what order. For example, if a piston needs to be installed next, generate an image showing its location and a voice command saying, 'Install the piston next.'"

[1768] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1769] Step 1:

[1770] The user wears a display device (head-mounted display).

[1771] Specifically, the user wears the display device on his / her head and the display is displayed within the field of vision. The display device has a built-in camera and microphone.

[1772] Step 2:

[1773] The user takes a picture of the engine parts with a camera and inputs into the microphone, "Which part should I install next?"

[1774] Specifically, the user points the camera at an engine part and takes a picture of it, while at the same time speaking into the microphone, "Which part should I install next?" The input data is video data and audio data.

[1775] Step 3:

[1776] The device captures video captured by the camera and audio recorded by the microphone.

[1777] In a specific operation, the terminal receives video data and audio data from the display device, the input data being the video data from the camera and the audio data from the microphone, and the output data being the captured video data and audio data.

[1778] Step 4:

[1779] The video captured by the device is analyzed using image analysis software to identify the type and location of engine parts.

[1780] Specifically, the device uses image analysis software to identify the type and location of engine parts from video data. The input data is the video data, and the output data is the type and location of the parts.

[1781] Step 5:

[1782] The voice captured by the device is converted into text data using voice recognition software.

[1783] Specifically, the terminal uses voice recognition software to convert voice data into text data, with the input data being voice data and the output data being text data.

[1784] Step 6:

[1785] The device inputs the image analysis results and voice recognition results into a generative AI model.

[1786] Specifically, the device inputs the image analysis results (type and position of parts) and voice recognition results (text data) into the generative AI model. The input data are the image analysis results and voice recognition results, and the output data is the input data for the generative AI model.

[1787] Step 7:

[1788] A generative AI model determines which parts to install next and in what order.

[1789] Specifically, the generative AI model analyzes the input data and determines the next part to be installed and the procedure for doing so. The input data is the results of image analysis and voice recognition, and the output data is the next part to be installed and the procedure for doing so.

[1790] Step 8:

[1791] The instructions determined by the generative AI model are output as text data.

[1792] Specifically, the instructions determined by the generative AI model are output to the terminal as text data. The input data is the next part to be installed and the procedure for installing it, and the output data is text data.

[1793] Step 9:

[1794] The terminal receives instructions from the generative AI model and displays an image showing the location of the parts on the display device's screen.

[1795] Specifically, the terminal receives instructions from the generative AI model and displays an image showing the position of the parts on the display of the display device. The input data is text data, and the output data is the image displayed on the display.

[1796] Step 10:

[1797] The device uses speech synthesis software to generate a voice message saying, "Next, install the piston," and communicates this to the user through the speaker.

[1798] In concrete terms, the terminal uses speech synthesis software to generate a voice message saying, "Next, install the piston," and transmits this message to the user through the speaker of the display device. The input data is text data, and the output data is the generated voice message.

[1799] (Application example 2)

[1800] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1801] In conventional assembly and repair work, workers had to spend a lot of time and effort trying to figure out which part to install next and the procedure for doing so. There was also a risk that workers would install the wrong part, which could have a negative impact on work efficiency and quality. This has led to a demand for improved work efficiency and quality.

[1802] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1803] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generating AI, a means for the generating AI to output instructions for the next work procedure to the worker as voice or image, a means for the generating AI to analyze the image and determine the next part to be installed and the procedure, a means for displaying an image showing the position of the part on the head mounted display, and a means for issuing voice instructions through a speaker. This enables the worker to accurately grasp the next part to be installed and the procedure in real time, thereby improving work efficiency and quality.

[1804] "Worker" means a person who assembles or repairs machines or vehicles in a factory or work site.

[1805] A "camera" is a device that captures images and records them as digital data.

[1806] A "microphone" is a device that records sound and saves it as digital data.

[1807] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[1808] "Input" refers to the input of data or information into a system.

[1809] "Output" refers to the system outputting the results of its processing.

[1810] A "means" is a method or device for achieving a specific purpose.

[1811] "Video" refers to visual information captured by a camera.

[1812] "Sound" refers to auditory information recorded by a microphone.

[1813] "Instructions" refers to commands to perform a particular action or task.

[1814] An "image" is digital data that represents visual information.

[1815] A "head-mounted display" is a display device that is worn on the head.

[1816] A "speaker" is a device for reproducing sound.

[1817] A "part" is an individual element that makes up a machine or vehicle.

[1818] A "procedure" is the order or method for performing a particular task.

[1819] As an embodiment of the present invention, we provide a system in which a factory robot is fitted with a head-mounted display and, when performing assembly work, uses generative AI to indicate in real time the next part to be installed and the procedure for doing so.

[1820] The server uses the following hardware and software:

[1821] Hardware: Camera, microphone, head-mounted display, speaker

[1822] Software: OpenCV (video capture and display), pyttsx3 (voice synthesis), transformers (generative AI model)

[1823] The server first uses a camera to capture the current assembly status and sends the video to the generation AI. The generation AI analyzes the video and audio and determines which parts should be installed next and in what order. It then displays an image showing the position of the parts on the head-mounted display and issues a voice command over the speaker, saying, "Install the piston next."

[1824] As a concrete example, when a robot is assembling an engine, a camera captures the current assembly status and sends a voice command such as "Which part should be installed next?" to the generation AI. The generation AI interprets this as "Install the piston next," displays an image showing the piston position on the head-mounted display, and issues a voice command over the speaker.

[1825] An example of a prompt is as follows:

[1826] "Please analyze the video showing the current assembly status and tell me which part should be installed next and in what order."

[1827] In this way, factory robots can efficiently carry out assembly work.

[1828] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1829] Step 1:

[1830] The server uses a camera to capture the current assembly status.

[1831] Input: Video data from the camera

[1832] Output: Captured video frames

[1833] How it works: The camera captures video in real time and acquires data frame by frame.

[1834] Step 2:

[1835] The server sends the captured video frames to the generation AI.

[1836] Input: Captured video frame

[1837] Output: Video data sent to the generation AI

[1838] What it does: Video frames are sent as digital data to a generative AI model.

[1839] Step 3:

[1840] The server uses a microphone to record the voice of the worker.

[1841] Input: Audio data from the microphone

[1842] Output: Recorded audio data

[1843] Specific operation: A microphone records the worker's voice in real time and stores it as digital data.

[1844] Step 4:

[1845] The server sends the recorded voice data to the generation AI.

[1846] Input: Recorded audio data

[1847] Output: Audio data sent to the generation AI

[1848] Specific operations: Audio data is sent as digital data to the generative AI model.

[1849] Step 5:

[1850] The generative AI analyzes video and audio data to determine the next part to be installed and the order in which it should be installed.

[1851] Input: Video data, audio data

[1852] Output: Text data of the next part to be installed and the procedure

[1853] Specific operation: The generative AI model analyzes video and audio and generates the appropriate parts and steps as text.

[1854] Step 6:

[1855] The server displays instructions from the generation AI on a head-mounted display.

[1856] Input: Text data of the next part to be installed and its procedure

[1857] Output: Image data displayed on the head-mounted display

[1858] Specific operation: Text data is converted into image data and displayed on a head-mounted display.

[1859] Step 7:

[1860] The server outputs instructions from the generation AI as audio through a speaker.

[1861] Input: Text data of the next part to be installed and its procedure

[1862] Output: Audio data output from the speaker

[1863] Specific operation: Convert text data into audio data and play it over a speaker.

[1864] In this way, the server instructs the worker on the next work procedure in real time, supporting efficient assembly work.

[1865] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1866] "Example 1"

[1867] In one embodiment of the present invention, a worker wears a head-mounted display. This display has a built-in camera and microphone, and an emotion engine. The worker starts a task, takes an image of the part with the camera of the head-mounted display, and records questions about the task with the microphone. These images and voices are input to the generation AI. The generation AI analyzes the images and voices and determines the next task procedure. Furthermore, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next task procedure based on the emotions. For example, if the emotion engine senses that the worker is stressed, it outputs an instruction to encourage the worker to take a break.

[1868] "Example 2"

[1869] In another embodiment of the present invention, a worker starts assembling an automobile engine, takes images of engine parts with a camera on a head-mounted display, and records a question with a microphone: "Which part should I install next?" These images and voices are input to a generative AI. The generative AI analyzes the images and voices and determines the next part to be installed and the order in which to install it. Furthermore, the emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on the emotions. For example, if the worker feels tired, the emotion engine outputs an instruction to encourage the worker to take a break.

[1870] The process flow of each embodiment will be described below.

[1871] "Example 1"

[1872] Step 1: The worker wears a head-mounted display, which has a built-in camera, microphone, and emotion engine.

[1873] Step 2: The worker begins the task, taking images of the part with the camera in the head-mounted display and recording questions about the task with the microphone.

[1874] Step 3: These images and audio are input into the generative AI.

[1875] Step 4: The generative AI analyzes the video and audio and determines the next steps.

[1876] Step 5: The emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on those emotions.

[1877] Step 6: For example, if the worker feels stressed, the emotion engine outputs an instruction to encourage the worker to take a break.

[1878] "Example 2"

[1879] Step 1: A worker begins assembling a car engine, taking images of the engine parts with a camera in a head-mounted display and recording a question with a microphone: "Which part should I install next?"

[1880] Step 2: These images and audio are input into the generative AI.

[1881] Step 3: The generative AI analyzes the video and audio to determine the next part to be installed and the order in which it should be installed.

[1882] Step 4: The emotion engine recognizes emotions from the worker's voice and adjusts the instructions for the next work step based on those emotions.

[1883] Step 5: For example, if a worker feels tired, the emotion engine outputs an instruction to encourage the worker to take a break.

[1884] Example 1

[1885] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1886] In conventional work support systems, it is sometimes difficult to give appropriate instructions on the next work procedure that a worker should perform. In addition, because the system gives uniform instructions without considering the worker's emotional state, it is difficult to reduce the worker's stress and fatigue. This reduces work efficiency and increases the possibility of work errors.

[1887] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1888] In this invention, the server includes a means for inputting the image captured by the camera held by the worker and the voice of the worker through the microphone to the generation AI, a means for the generation AI to output instructions for the next work procedure to the worker as voice or image, and a means for the emotion engine to recognize emotions from the voice of the worker and adjust the instructions for the next work procedure based on the emotions. This makes it possible to give appropriate instructions for work procedures that take into account the emotional state of the worker, which is expected to improve work efficiency and reduce work errors.

[1889] "Worker" refers to a person who performs a particular task.

[1890] "Camera" refers to a device for capturing images.

[1891] "Microphone" refers to a device for recording sound.

[1892] "Generative AI" refers to artificial intelligence that analyzes input data and generates appropriate output.

[1893] "Input" refers to entering data into a system.

[1894] "Output" refers to sending data out of the system.

[1895] An "emotion engine" is a system that recognizes emotions from voice and video, and generates appropriate instructions based on that information.

[1896] "Video" refers to visual information captured by a camera.

[1897] "Audio" refers to auditory information recorded by a microphone.

[1898] "Work procedure" refers to the specific steps or methods for performing a particular task.

[1899] "Instructions" refer to information intended to encourage a particular action.

[1900] A "system" refers to a series of devices or programs in which multiple elements work together.

[1901] This invention is a system in which a worker wears a head-mounted display and inputs video and audio to a generative AI via the display's built-in camera and microphone. The generative AI analyzes the video of the parts the worker should handle and the worker's audio (e.g., "Which part should I install next?") and determines the next work procedure. The instructions are communicated to the worker via the display and speaker of the head-mounted display.

[1902] Hardware and Software Used

[1903] Hardware: Head-mounted display (built-in camera, microphone, display, speaker)

[1904] Software: Generative AI models, emotion engines

[1905] Specific operation of the system

[1906] Subject: Server

[1907] The server receives video and audio data transmitted from the head-mounted display worn by the worker. This data is collected through the built-in camera and microphone. The server analyzes the received video data and identifies the part that the worker should handle. It also analyzes the audio data to understand the worker's question (e.g., "Which part should I install next?"). Based on the results of this analysis, the server determines the next work procedure and conveys those instructions to the worker via the display and speakers of the head-mounted display.

[1908] Subject: Terminal

[1909] The terminal (head-mounted display) is a device worn by the worker. The terminal uses a built-in camera and microphone to collect images of parts within the worker's field of vision and the worker's voice. The collected data is sent to the server in real time. When the terminal receives instructions from the server, it displays the instructions on the display and conveys them to the worker by voice through the speaker.

[1910] Subject: User

[1911] The user (worker) wears the head-mounted display while performing their work. As the user handles parts, the built-in camera captures the image, and the microphone records any questions about the work. When the user asks, "Which part should I install next?", the voice data is sent to the server. After receiving instructions from the server, the user checks the next work procedure through the display and speaker of the head-mounted display.

[1912] Examples

[1913] Example 1: Parts installation

[1914] When a user asks, "Which part should I attach next?", the microphone in the head-mounted display records the voice and sends it to the server. The server analyzes the video and audio data to identify the next part to be attached. For example, if the server instructs, "Attach the screw next," the instruction is displayed on the display of the head-mounted display and communicated by voice through the speaker. The user follows the instruction to attach the screw.

[1915] Example 2: Emotion-recognition-based break instruction

[1916] If a user says "I'm tired" while working, the microphone in the head-mounted display records the voice and sends it to the server. The emotion engine recognizes the user's stress from the voice. The emotion engine generates an instruction to encourage the user to take a break, and the server sends the instruction to the head-mounted display. The device displays the instruction "Let's take a short break" on the display and conveys it by voice through the speaker. The user follows the instruction and takes a break.

[1917] Examples of prompt statements

[1918] "If a worker wears a head-mounted display and asks what the next step is, please explain how the generative AI model would determine the next step and communicate it to the worker."

[1919] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1920] Step 1:

[1921] The user wears a head-mounted display.

[1922] Specifically, the user wears the head-mounted display and checks that the built-in camera and microphone are working properly, enabling the collection of video and audio.

[1923] Step 2:

[1924] The device collects video and audio.

[1925] Specifically, the device's built-in camera captures images of parts that are within the user's field of vision, and the built-in microphone records the user's voice. For example, if the user asks, "Which part should I install next?", that voice is recorded.

[1926] Input: Images of parts in the user's field of vision, user's voice

[1927] Output: Collected video and audio data

[1928] Step 3:

[1929] The terminal transmits the collected data to the server.

[1930] Specifically, the device transmits collected video and audio data in real time to the server, which then prepares the data for analysis.

[1931] Input: Collected video and audio data

[1932] Output: Data sent to the server

[1933] Step 4:

[1934] The server parses the data.

[1935] Specifically, the server analyzes the received video data to identify the part the user is handling. It also analyzes the voice data to understand the user's question. For example, if the user asks, "Which part should I install next?", the server analyzes and understands the question.

[1936] Input: Video and audio data sent to the server

[1937] Output: Analysis results (part identification, question understanding)

[1938] Step 5:

[1939] The server decides the next step to take.

[1940] Specifically, the server determines the next step based on the analysis results. For example, if the user determines that the part to be installed is a screw, the server generates the instruction "Install the screw next."

[1941] Input: Analysis results (part identification, understanding of question content)

[1942] Output: Next step instructions

[1943] Step 6:

[1944] The server sends the instruction to the terminal.

[1945] Specifically, the server sends instructions for the determined work procedure to the terminal, including the specific work that the user should do next.

[1946] Input: Next step instructions

[1947] Output: Instructions sent to the terminal

[1948] Step 7:

[1949] The terminal conveys the instructions to the user.

[1950] Specifically, the terminal conveys the instructions received from the server to the user. Specifically, the instructions are displayed on the head-mounted display and conveyed by voice through the speaker. For example, the instruction "Next, install the screws" is displayed on the display and conveyed by voice.

[1951] Input: Instructions sent to the terminal

[1952] Output: Instructions given to the user

[1953] (Application example 1)

[1954] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1955] In conventional factory work, workers had to refer to the manual or ask their superiors to confirm the next work procedure, ...

Claims

1. A means for mounting a display device held by an operator; A means for acquiring an image through a camera and a voice of a worker through a microphone; A means for analyzing the acquired image and identifying the type and position of the part; A means for converting the acquired voice into text data; A means for inputting the image analysis result and the voice recognition result into a generative AI model; A means for determining the next part to be installed and the procedure thereof by the generative AI model to which the image analysis result and the voice recognition result have been input; A means for outputting the instruction determined by the generative AI model as text data; means for displaying an image showing the position of the part on a display of a display device; A means for generating instructions in voice form and communicating the instructions to the worker through a speaker; A system including:

2. 2. The system of claim 1, wherein the image is an image of a part associated with an assembly or repair operation of a machine, vehicle, or the like.

3. The system according to claim 1 , wherein the voice is a voice asking a question about the worker's work.

Images