System

Abstract

A system is provided.SOLUTION: A system includes a means for receiving an image photographed by a user, a means for analyzing the image to extract feature information, a means for collating the extracted feature information with information in a database, and a means for transmitting a collation result to a terminal of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In base station construction work, identifying the materials being used is difficult and time-consuming for inexperienced employees. This reduces on-site work efficiency, making it difficult for new employees and transferred employees to contribute immediately. Furthermore, incorrect material identification risks inaccurate decisions about the installation of additional antennas and radio equipment. To solve these issues, a system is needed that can quickly and accurately identify materials and improve on-site work efficiency. [Means for solving the problem]

[0005] The present invention is a system that includes a means for receiving images taken by a user, a means for analyzing the images and extracting feature information, a means for comparing the extracted feature information with information in a database, and a means for transmitting the comparison results to the user's device. This allows for rapid analysis of images of components taken on-site and automatic identification of specific product names and model numbers. Furthermore, by providing a communication application for transmitting images taken by the user, a means for displaying detailed component information based on feature information, a means for using an AI module for comparison, and an interface for displaying the comparison results on the user's device, specific component information can be provided in a short time, allowing even new employees or newly transferred employees to contribute immediately to the work. This is expected to improve efficiency and accuracy on-site, leading to improved quality and reduced workloads.

[0006] "User" refers to the person operating the system and taking and transmitting images.

[0007] "Terminal" refers to a communication device operated by a user (e.g., smartphone, tablet).

[0008] "Server" refers to a computer system that analyzes received images and compares them with a database.

[0009] "Image" refers to visual data such as a photograph or drawing of an item that a user takes and sends to the system.

[0010] "Feature information" refers to information used to identify shapes, text, colors, etc. extracted by image analysis.

[0011] "Database" refers to data storage within the system that stores detailed information about components.

[0012] "Matching" refers to the process of comparing extracted feature information with information in a database to identify matching components.

[0013] "Communication application" refers to software that allows users to send images (e.g., LINE, WhatsApp).

[0014] "AI module" refers to an artificial intelligence system (e.g., TensorFlow model) that analyzes images and extracts feature information.

[0015] "Interface" refers to the screen or application display area on the terminal that displays the matching results to the user.

[0016] An "antenna" refers to a radio wave transmitting and receiving device installed at a base station for wireless communication.

[0017] "Radio" refers to communication equipment for transmitting and receiving voice and data wirelessly.

[0018] "Concrete pillar" refers to a columnar member used as part of the structure of a base station. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system for quickly and accurately identifying materials in base station construction work, allowing even new employees and transferred employees to contribute immediately to the work.

[0041] System Overview

[0042] The system receives images taken by the user, analyzes them to extract characteristic information, compares them with a database, and returns the results to the user, allowing the user to easily identify components.

[0043] Program processing flow

[0044] 1. Taking and sending images

[0045] Users take pictures of components (e.g., antennas, radios, and concrete poles) at base station sites using their smartphone cameras, and then use communication applications (e.g., LINE, WhatsApp) to send the images to the system.

[0046] 2. Receiving and temporarily storing images

[0047] The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this point, the device converts the image into the appropriate format (e.g., JPEG, PNG) and prepares it for transmission to the server.

[0048] 3. Image submission and analysis

[0049] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0050] 4. Matching feature information

[0051] The server compares the feature information extracted by the AI ​​module with the component information in the database, retrieves detailed information such as the product name and model number of the relevant component from the database, and selects the result with the best match.

[0052] 5. Generating and sending matching results

[0053] The server generates a match result to send to the user's device, encodes it as JSON data, and sends this data to the user's device as an HTTP response.

[0054] 6. Display of matching results

[0055] The device decodes the response received from the server and displays the matching result in a format that is easy for the user to understand. The user can then confirm the recognition result (e.g., "This is an antenna with model number XX") on the device.

[0056] Specific examples

[0057] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[0058] 2. The user sends the image to the system via a communication application.

[0059] 3. The device receives the image, converts it into the appropriate format, and sends it to the server.

[0060] 4. The server receives the image and passes it to the AI ​​module for analysis.

[0061] 5. The AI ​​module extracts the antenna's characteristic information and compares it with a database.

[0062] 6. The server identifies the product name and model number of the matching antenna and generates a match.

[0063] 7. The server sends the matching results to the user's device.

[0064] 8. The device displays the matching results to the user and notifies them that the antenna is model number XX.

[0065] Through this process, the system enables users to quickly identify components on-site, improving the efficiency and accuracy of on-site operations.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The user takes pictures of components (e.g., antennas, radios, and concrete poles) at the base station site using their smartphone camera.

[0069] Step 2:

[0070] The user opens a communication application (e.g., LINE, WhatsApp), selects the captured image, and presses the "Send" button to send it to the system.

[0071] Step 3:

[0072] The device receives the image via the communication application, temporarily stores the image data in its internal memory, and converts the image to an appropriate format (e.g., JPEG, PNG).

[0073] Step 4:

[0074] The device generates an HTTP POST request to send the saved image data to the server, and sends it to the server's image reception endpoint.

[0075] Step 5:

[0076] The server receives the HTTP request, extracts the image data from the request body, and temporarily stores it.

[0077] Step 6:

[0078] The server passes the stored image to the AI ​​module and begins image analysis.

[0079] Step 7:

[0080] The AI ​​module (in the server) analyzes the image and extracts feature information such as shape, text, and color.

[0081] Step 8:

[0082] Based on the feature information extracted by the AI ​​module, the server generates an SQL query to match it with component information in the database.

[0083] Step 9:

[0084] The server runs an SQL query against the database to obtain detailed information such as the product name and model number of the relevant part.

[0085] Step 10:

[0086] The server selects the best match, encodes the recognition result in JSON format, stores it in an HTTP response, and sends it to the user's device.

[0087] Step 11:

[0088] The device receives the response from the server and decodes the JSON data.

[0089] Step 12:

[0090] The device updates the GUI to display the recognition result (e.g., "This is an antenna with model number XX") to the user.

[0091] Step 13:

[0092] The user checks the recognition results displayed on the terminal and, if necessary, proceeds with further investigation or response.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] Identifying materials in base station construction work requires engineers with specialized knowledge, and it is difficult for new employees or transferred employees to quickly and accurately identify materials on-site. This can reduce work efficiency and affect accuracy. Furthermore, there is a risk of frequent mistakes and rework on-site, so a solution to these issues is needed.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes a device that receives images taken by a user, an analysis device that analyzes the images and extracts feature information, a comparison device that compares the extracted feature information with information in a database, and a communication device that transmits the comparison results to the user's device, thereby enabling the user to quickly and accurately identify components based on images taken at the site.

[0098] A "user" is a person who uses the system to identify components.

[0099] An "image" is a file that represents visual information of an object photographed by a user.

[0100] A "device" is a device for receiving images taken by a user.

[0101] An "analysis device" is a device that analyzes a received image and extracts feature information.

[0102] "Feature information" is discernible data such as shape, text, and color extracted from an image.

[0103] A "database" is a digital storage device that stores information to be matched with characteristic information.

[0104] A "matching device" is a device that matches the extracted feature information with information in a database.

[0105] A "communication device" is a device for transmitting the collation results to the user's device.

[0106] "Communication software" is a program for transmitting images taken by the user.

[0107] The "display device" is a device for displaying detailed information about a component based on its characteristic information.

[0108] The "system" is a set of devices and software that identifies components based on images taken by the user.

[0109] This invention relates to a system for quickly and accurately identifying components in base station construction work. The system analyzes images taken by users on-site, compares them with a database, and provides information on the relevant components. The specific system configuration, the hardware and software used, and the data processing method are described in detail below.

[0110] System configuration

[0111] The system mainly consists of the following components:

[0112] 1. Your device

[0113] A mobile device such as a smartphone or tablet is used, which has a camera and communication software (e.g., LINE, WhatsApp) installed.

[0114] 2. Analysis device

[0115] It uses an AI module located on the server, specifically consisting of hardware and software (e.g., TensorFlow, PyTorch) for executing image recognition algorithms.

[0116] 3. Database

[0117] A central database that stores component information, including detailed information such as component product name, model number, shape, and color.

[0118] 4. Communications Equipment

[0119] It includes the network infrastructure (e.g. HTTP server, RESTful API) for communication between the server and the user's device.

[0120] Specific program processing and usage

[0121] 1. Taking and sending images

[0122] The user takes an image of the component at the base station site using the smartphone camera, then launches the communication software on the smartphone and sends the image to the system.

[0123] 2. Receiving and temporarily storing images

[0124] The device receives the image via the communication software, temporarily stores it in its internal memory, and converts it into the appropriate format (e.g., JPEG, PNG) as needed.

[0125] 3. Image submission and analysis

[0126] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0127] 4. Matching feature information

[0128] The server compares the feature information extracted by the AI ​​module with the component information in the database, selects the most matching result, and obtains detailed information.

[0129] 5. Generating and sending matching results

[0130] The server generates a match result to send to the user's device, encodes it as JSON data, and sends it to the user's device as an HTTP response.

[0131] 6. Display of matching results

[0132] The device decodes the response received from the server and displays the matching results in a format that is easy for the user to understand, such as a message like "This is an antenna with model number XX."

[0133] Specific examples

[0134] For example:

[0135] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[0136] 2. The user selects an image in the LINE app and sends it to the system.

[0137] 3. The device receives the image, converts it to JPEG format, and saves it to its internal memory.

[0138] 4. The device sends the saved image to the server.

[0139] 5. The server receives the image and passes it to the AI ​​module for analysis.

[0140] 6. The AI ​​module extracts features such as antenna shape, text, and color and compares them with a database.

[0141] 7. The server identifies the product name and model number of the best matching antenna and generates the results in JSON format.

[0142] 8. The server sends the generated results to the user's device.

[0143] 9. The device analyzes the results received and displays to the user, "This is an antenna with model number XX."

[0144] Example prompts for generative AI models

[0145] Possible prompts include:

[0146] "You are developing a system to identify images of base station antennas. Follow these steps to describe a program that will analyze the image, match it against a database, and return the results."

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Program processing steps

[0149] Step 1: User takes and submits image

[0150] The user takes an image of the components (e.g., antenna, radio, concrete pole) at the base station site using the smartphone camera. When taking the photo, the camera is adjusted so that the components are clearly visible. After taking the photo, the user launches a communication application on the smartphone (e.g., LINE, WhatsApp), selects the image, and sends it to the system.

[0151] Input: Image taken by the camera

[0152] Output: Image data sent to the system via a communication application

[0153] Step 2: Receiving and temporarily saving images on the device

[0154] The device receives images sent by the user via a communication application. The received image data is temporarily stored in internal memory. When saving, the device checks whether the image is in an appropriate format (e.g., JPEG, PNG), and if it is not, converts it to an appropriate format. It also checks that the image resolution is within a range suitable for analysis.

[0155] Input: Image data sent from a communication application

[0156] Output: Image data in the appropriate format, stored in the internal memory

[0157] Step 3: Sending images from the device to the server and analyzing them

[0158] The device generates a request to send pre-prepared image data to the server. This request also includes metadata (e.g., date and time of transmission, user ID). Once the request is sent, the server receives the request and saves the image data in a folder for internal processing. The server then passes the saved image data to the AI ​​module, which starts the image analysis process. The AI ​​module first pre-processes the image, removing noise and filtering, and then extracts feature information such as shape, text, and color.

[0159] Input: Image data stored in the internal memory

[0160] Output: Feature information extracted by the AI ​​module

[0161] Step 4: Server checks the feature information

[0162] The server compares the feature information extracted by the AI ​​module with the component information in the database. It uses a high-speed search algorithm (e.g., KD-Tree, Locality-Sensitive Hashing) to find the best match. It retrieves the product name, model number, and related details of the component from the database. If there are multiple candidates, it selects the best one.

[0163] Input: Feature information provided by the AI ​​module

[0164] Output: Component information extracted from the database (product name, model number, detailed information)

[0165] Step 5: Server generates and sends match results

[0166] The server generates a matching result to send to the user's device. This matching result is encoded as JSON data and includes necessary information (e.g., component product name, model number, and links to related documents). The server generates an HTTP response, embeds the JSON data in it, and sends it to the user's device.

[0167] Input: Material information extracted from the database

[0168] Output: Match results encoded in JSON format

[0169] Step 6: Display the match results on your device

[0170] The device analyzes the HTTP response received from the server and obtains the decoded JSON data. The device displays this data in a format that is easy for the user to understand. Specifically, the device displays a message such as "This is an antenna with model number XX" on the user screen, along with a link to more detailed information. The user can confirm this and check the detailed information as needed.

[0171] Input: JSON data of the match result sent from the server

[0172] Output: Material information displayed to the user

[0173] (Application example 1)

[0174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0175] In factories, the task of identifying parts is typically manual and often time-consuming and labor-intensive. Furthermore, it is difficult for inexperienced workers to accurately grasp the information on identified parts, which can lead to problems due to misidentification. This can reduce work efficiency and negatively impact the productivity of the entire process. To solve these problems, a system that can quickly and effectively identify and manage parts is needed.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0177] In this invention, the server includes means for receiving images taken by a user, means for analyzing the images and extracting feature information, means for comparing the extracted feature information with information in a database, means for transmitting the comparison result to the user's terminal, means for the robot to take an image of a component, means for saving the image data in a built-in computer and transmitting the image data to an analysis module after converting its format, means for extracting feature information such as shape, text, and color using the image analysis module, and means for reporting component information to a management system using the extracted feature information. This enables the robot to automatically identify components in a factory and report information to the management system in real time.

[0178] "User" refers to a person who uses this system to take images and transmit data.

[0179] "Image" refers to still image data captured by a camera.

[0180] "Means for receiving" refers to devices or software that have the function of receiving and storing captured image data.

[0181] "Means for analyzing" refers to algorithms or software for extracting feature information from received image data.

[0182] "Feature information" refers to data such as shape, text, and color obtained through image analysis.

[0183] "Database" refers to a collection of information that stores detailed information about components.

[0184] "Means for matching" refers to a device or software that has the function of comparing extracted feature information with information in a database to find a match.

[0185] The "matching result" refers to the specific information of the component obtained by the matching means.

[0186] "Terminal" refers to an electronic device used by a user, such as a smartphone or computer.

[0187] "Robot" refers to a self-propelled mechanical device used to identify and manage parts within a factory.

[0188] "On-board computer" refers to a data processing device installed in a robot.

[0189] "Image analysis module" refers to a program or software for analyzing image data and extracting feature information.

[0190] "Management system" refers to a system for managing and displaying information about components within a factory.

[0191] This invention relates to a system in which a robot automatically identifies parts in a factory and reports the information to a management system in real time. This system implements a series of processes: receiving images taken by a user, analyzing them, extracting characteristic information, comparing them with a database, and sending the results. Furthermore, the system also includes functions for the robot to take images of parts, save the data in its built-in computer, send it to an analysis module, extract characteristic information, and report the part information to the management system.

[0192] The server has a means for receiving images taken by the user. For example, when a user takes an image with a smartphone and sends it via a communication application, the server receives the image and temporarily stores it.

[0193] Next, the server has a means to analyze the image and extract feature information. Here, an AI-based image analysis module is used, for example, the OpenCV library. This module extracts feature information such as shape, text, and color from the image data.

[0194] The server has a means for comparing the extracted feature information with information in a database. The database stores component information, and the server compares it with the information that best matches it.

[0195] The server has a means to send the matching results to the user's device. This means sends JSON format data to the user's device using an HTTP request. The user can check the results on their smartphone or other device.

[0196] The robot also has a means of taking images of the parts as it moves. The images are stored in the robot's onboard computer and converted to the appropriate format as needed. This computer then sends the image data to the analysis module, which then begins the image analysis.

[0197] The onboard computer uses an image analysis module to extract feature information such as shape, text, and color. Again, the OpenCV library can be used. The extracted feature information is reported to the management system.

[0198] For example, to identify new machine parts in a factory, a robot patrols a parts warehouse and takes pictures. The images are sent to a system, where they are analyzed and the results are reported to a management system. Workers can use this information to work more efficiently.

[0199] An example of a prompt for the generative AI model is, "Extract the feature information of the component from the image and compare it with the database. Identify the product name and model number of the corresponding component and display them to the user." Using this prompt, the AI ​​module can efficiently analyze the feature information and identify the component.

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1:

[0202] The user takes an image of a component using a smartphone or robot camera. The input is real-time image data obtained by the camera, and the output is an image file (e.g., JPEG, PNG).

[0203] Step 2:

[0204] The image taken by the user or robot is sent to the server via a communication application. Specifically, the user sends the image from their smartphone using a communication application (e.g., LINE, WhatsApp), and the robot similarly sends it to the server using its built-in communication module. The purpose of this step is for the image data to reach the server. The input is the captured image file, and the output is the sent image data.

[0205] Step 3:

[0206] The server temporarily stores the received image data. This storage process includes converting the image data to the appropriate file format (e.g., converting JPEG to PNG). The input is the raw image data received by the server, and the output is image data that has been appropriately saved in the temporary storage location, such as memory or storage.

[0207] Step 4:

[0208] The server passes the saved image data to the analysis module and begins image analysis. The analysis uses libraries such as OpenCV to extract feature information such as shape, text, and color from the image. The input is the temporarily saved image data, and the output is the extracted feature information. Specific operations include image filtering, text recognition, and shape analysis using OpenCV.

[0209] Step 5:

[0210] The server compares the extracted feature information with information in the database. Here, a comparison is made with pre-registered component information to identify the component information that best matches. The input is the extracted feature information and component information in the database, and the output is the specific information of the component that best matches (e.g., product name, model number).

[0211] Step 6:

[0212] The server generates matching results and sends them to the user's device. The generated matching results are encoded in a format such as JSON and sent to the user's device as an HTTP request. The input is information about the identified components, and the output is the encoded matching result data.

[0213] Step 7:

[0214] The terminal decodes the matching results received from the server and displays them to the user. The user can check the results on their smartphone or the management system. The input is the matching result data sent from the server, and the output is information displayed in a format that is easy for the user to understand. Specific operations include parsing JSON data and displaying the information on the UI.

[0215] Step 8:

[0216] The robot then patrols the parts in the factory, taking new images of the parts and sending them to the system. In this step, the robot's built-in computer stores the image data, passes it to an analysis module to extract feature information, and reports it to the management system. The input is the new image data taken by the robot, and the output is the part information reported to the management system.

[0217] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0218] This invention is a system for quickly and accurately identifying components in base station construction work, and aims to improve the user experience by recognizing the user's emotions and providing feedback based on those emotions. This allows even new employees and transferred employees to contribute to the work as an immediate asset.

[0219] System Overview

[0220] The system receives images taken by the user, analyzes them to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback on the results, achieving a series of steps. This allows the user to easily identify components, and the system recognizes and provides feedback on the user's emotions during the process, reducing stress for the user.

[0221] Program processing flow

[0222] 1. Taking and sending images

[0223] Users take pictures of components (e.g., antennas, radios, and concrete poles) at base station sites using their smartphone cameras, and then use communication applications (e.g., LINE or WhatsApp) to send the images to the system.

[0224] 2. Receiving and temporarily storing images

[0225] The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this point, the device converts the image into the appropriate format (e.g., JPEG, PNG) and prepares it for transmission to the server.

[0226] 3. Image submission and analysis

[0227] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0228] 4. Matching feature information

[0229] The server compares the feature information extracted by the AI ​​module with the component information in the database, retrieves detailed information such as the product name and model number of the relevant component from the database, and selects the result with the best match.

[0230] 5. Emotion Recognition and Feedback

[0231] In parallel with image matching, the server runs an emotion engine that extracts emotional information from the user's facial image, voice, etc. The emotion engine recognizes the user's emotions (e.g., stress, anxiety, joy, etc.) and generates analysis results.

[0232] 6. Generating and transmitting emotion-based matching results

[0233] The server generates a match result adjusted based on the analysis results of the emotion engine and sends it to the user's device, which also includes a feedback message according to the user's emotion.

[0234] 7. Displaying matching results and feedback

[0235] The device decodes the response received from the server and displays the matching result and feedback in a format that is easy for the user to understand. The user can see the recognition result (e.g., "This is an antenna with model number XX") on the device along with a feedback message corresponding to the user's emotion.

[0236] Specific examples

[0237] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[0238] 2. The user sends the image to the system via a communication application.

[0239] 3. The device receives the image, converts it into the appropriate format, and sends it to the server.

[0240] 4. The server receives the image and passes it to the AI ​​module for analysis.

[0241] 5. The AI ​​module extracts the antenna's characteristic information and compares it with a database.

[0242] 6. The server identifies the product name and model number of the matching antenna, and in parallel the emotion engine recognizes the user's emotion.

[0243] 7. The server generates feedback based on the matching results and the user's emotions and sends it to the user's device as JSON format data.

[0244] 8. The device displays the matching results and feedback to the user, notifying them with a message such as "This is an antenna with model number XX" and "Good job, almost done."

[0245] Through this process, the system can help users quickly identify parts on-site, while also recognizing users' emotions and providing appropriate feedback to improve the user experience.

[0246] The processing flow will be explained below.

[0247] Step 1:

[0248] The user takes pictures of components (e.g., antennas, radios, and concrete poles) at the base station site using the smartphone camera.

[0249] Step 2:

[0250] The user opens a communication application (e.g., LINE, WhatsApp), selects the captured image, and presses the "Send" button to send it to the system.

[0251] Step 3:

[0252] The device receives the image via the communication application, temporarily stores the image data in its internal memory, and converts the image to an appropriate format (e.g., JPEG, PNG).

[0253] Step 4:

[0254] The device generates an HTTP POST request to send the saved image data to the server, and sends it to the server's image reception endpoint.

[0255] Step 5:

[0256] The server receives the HTTP request, extracts the image data from the request body, and temporarily stores it.

[0257] Step 6:

[0258] The server passes the stored image to the AI ​​module and begins image analysis.

[0259] Step 7:

[0260] The AI ​​module (in the server) analyzes the image and extracts feature information such as shape, text, and color.

[0261] Step 8:

[0262] Based on the feature information extracted by the AI ​​module, the server generates an SQL query to match it with component information in the database.

[0263] Step 9:

[0264] The server runs an SQL query against the database to obtain detailed information such as the product name and model number of the relevant part.

[0265] Step 10:

[0266] The server generates the matching results and simultaneously runs an emotion engine that extracts emotion information from the user's facial image, voice, etc.

[0267] Step 11:

[0268] The emotion engine (in the server) recognizes the user's emotions (e.g., stress, anxiety, joy, etc.) and generates the analysis results.

[0269] Step 12:

[0270] The server generates an adjusted matching result based on the analysis results of the emotion engine and sends it to the user's device, which also includes a feedback message according to the user's emotion.

[0271] Step 13:

[0272] The device receives the response from the server and decodes the JSON data.

[0273] Step 14:

[0274] The terminal displays the recognition result (e.g., "This is an antenna with model number XX") to the user, as well as a feedback message according to the user's emotion (e.g., "Good job, you're almost done").

[0275] Step 15:

[0276] The user checks the recognition results and feedback messages displayed on the terminal, and if necessary, proceeds with further investigation or response.

[0277] This specific step allows the system to help users quickly identify components in the field, and also recognizes the user's emotions and provides appropriate feedback.

[0278] Example 2

[0279] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0280] In base station construction work, there is a need for a method that allows even new employees or newly transferred employees to quickly and accurately identify components. There is also a need for a feedback function to reduce the stress and anxiety of users who work long hours. There has not been a system that meets these needs, so there is a need to solve these issues.

[0281] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0282] In this invention, the server includes means for receiving an image taken by a user, means for analyzing the image and extracting feature information, means for comparing the extracted feature information with information in a database, means for recognizing the user's emotion, means for generating feedback based on the emotion recognition result, and means for transmitting the generated feedback to the user's terminal. This allows the user to easily identify components and receive appropriate feedback regarding changes in emotion during the process, thereby improving work efficiency and the user experience.

[0283] The "means for receiving images taken by a user" refers to a device or method for acquiring image data taken by a user with a photographing device such as a smartphone through communication.

[0284] "Means for analyzing images and extracting feature information" refers to a device or method that analyzes captured image data using machine learning algorithms or image processing techniques to extract features such as shape, text, and color within the image.

[0285] The "means for matching with information in a database" refers to a device or method that searches for items that match information in a pre-registered database based on the extracted characteristic information and generates a matching result.

[0286] "Means for recognizing user emotions" refers to a device or method for analyzing a user's facial expressions and tone of voice from image or audio data, etc., to detect the user's emotional state (e.g., stress, anxiety, joy, etc.).

[0287] The "means for generating feedback based on emotion recognition results" refers to a device or method for creating an appropriate feedback message according to the recognized emotional state of the user and notifying the user of the appropriate feedback message.

[0288] The "means for transmitting the generated feedback to the user's terminal" refers to a device or method for transmitting the generated feedback message from the server to the user's terminal such as a smartphone or tablet via data communication.

[0289] This invention is a system for quickly and accurately identifying components in base station construction work. It aims to improve the user experience by recognizing the user's emotions and providing feedback based on those emotions. This allows even new employees and newly transferred employees to contribute immediately to the work. The system receives images taken by the user, analyzes the images to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback based on the results.

[0290] The specific configuration of the system is as follows.

[0291] 1. Receiving images

[0292] The user uses the smartphone camera to take an image of the component at the base station site. The captured image is then sent to the system using a communication application (e.g., LINE or WhatsApp). The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this time, the device converts the image into an appropriate format (e.g., JPEG or PNG).

[0293] 2. Sending images and preparing for analysis

[0294] The device generates a request to send the stored image data to the server and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. This AI module performs image analysis using a deep learning framework (e.g., TensorFlow, PyTorch).

[0295] 3. Extraction of feature information

[0296] The server uses an AI module to extract feature information such as shape, text, and color from the image, which is then used for database matching.

[0297] 4. Database Matching

[0298] The server compares the extracted feature information with the registered information in the database. The database is built using a common RDBMS (e.g., MySQL or PostgreSQL) and stores detailed information such as the product name and model number of the component. The server selects the most matching result and temporarily stores it.

[0299] 5. Performing Emotion Recognition

[0300] In parallel, the emotion engine extracts emotional information from the user's facial image, voice, and other data. The emotion engine performs emotion analysis using, for example, OpenCV or a natural language processing (NLP) model. The recognized emotional data (e.g., stress level, anxiety level, happiness level) is saved.

[0301] 6. Producing adjusted matching results

[0302] The server adds a feedback message to the matching result based on the emotion recognition results. For example, if the user is feeling stressed, it generates a message such as "Good job, you're almost done." This data is converted into JSON format and sent to the user's device.

[0303] 7. Viewing Results and Feedback

[0304] The device receives the response from the server and displays the analysis results and feedback on the user interface. The user can check the recognition result (e.g., "This is an antenna with model number XX") and the emotional feedback message on the device.

[0305] Specific examples

[0306] A user takes a picture of the newly installed antenna of the base station with their smartphone.

[0307] The user uses a communication application to send the image to the system.

[0308] The device receives the image, converts it into the appropriate format, and sends it to the server.

[0309] The server receives the images and analyzes them using an AI module.

[0310] The AI ​​module extracts characteristic information and compares it with a database.

[0311] The server identifies the matching component information, and in parallel, the emotion engine recognizes the user's emotion.

[0312] The server generates adjusted matching results and emotional feedback and sends them to the device.

[0313] The device will display the results, along with a message such as "This is an antenna with model number XX" and "Thank you for your hard work, it's almost done."

[0314] Prompt Sentence Examples

[0315] "I would like to develop a system that analyzes and identifies images of components taken at the base station's work site and returns the results. It also needs a function that recognizes the user's emotions and provides feedback. Please explain in detail the program flow for such a system."

[0316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0317] Step 1:

[0318] The user uses the smartphone camera to take pictures of components (e.g., antennas, radios, and concrete poles) at the base station site. The input is the image taken by the user with the camera, and the output is an image file saved on the smartphone. Specifically, the user launches the smartphone's camera app, frames the component, and presses the capture button.

[0319] Step 2:

[0320] The user sends the captured image to the system using a communication application (e.g., LINE or WhatsApp). The input is an image file stored on the smartphone, and the output is image data sent through the communication application. Specifically, the user opens the communication application, displays the chat screen, attaches the image, and presses the send button.

[0321] Step 3:

[0322] The device receives images via a communication application and temporarily stores the image data in its internal memory. The input is the image data received from the communication application, and the output is an image file stored in a temporary storage folder. Specifically, the device receives an application notification, downloads the image data, and stores it in a temporary storage folder (e.g., / tmp / images).

[0323] Step 4:

[0324] The device converts the received image into an appropriate format (e.g., JPEG, PNG) and generates a request to send to the server. The input is the image file saved in the temporary storage folder, and the output is the image data sent to the server as an HTTP POST request. Specifically, the device checks the image format, performs conversion processing if necessary, and generates an HTTP POST request.

[0325] Step 5:

[0326] The server passes the received image data to the AI ​​module and begins image analysis. The input is the image data received as an HTTP POST request, and the output is the image data waiting to be analyzed. Specifically, the server obtains the image data via the file reception API and passes it to the AI ​​module.

[0327] Step 6:

[0328] The server uses an AI module to extract feature information such as shape, text, and color from images. The input is the image data passed to the AI ​​module, and the output is the extracted feature information. Specifically, the server runs an AI module (e.g., TensorFlow, PyTorch) and applies an image analysis algorithm to extract feature information.

[0329] Step 7:

[0330] The server compares the extracted feature information with the component information in the database. The input is the extracted feature information, and the output is the component information as a comparison result. Specifically, the server generates a database query and searches a database such as MySQL or PostgreSQL to obtain the most matching component information.

[0331] Step 8:

[0332] The server also uses an emotion engine to recognize the user's emotions. The input is the user's facial image and voice data, and the output is the recognized emotion data. Specifically, the server runs the emotion engine (e.g., OpenCV, NLP model) to extract emotion information from the facial image and voice.

[0333] Step 9:

[0334] The server generates an adjusted matching result and a feedback message based on the emotion recognition result. The input is the component information as the matching result and the recognized emotion data, and the output is the adjusted matching result and a feedback message. Specifically, the server creates a feedback message based on the emotion data and converts it into JSON format.

[0335] Step 10:

[0336] The server sends the generated feedback message to the user's device. The input is the matching result and feedback message converted to JSON format, and the output is the data sent to the user's device as an HTTP response. Specifically, the server sends the JSON data through an API endpoint.

[0337] Step 11:

[0338] The terminal receives the response from the server and displays the analysis results and feedback on the user interface. The input is the JSON data received from the server, and the output is the recognition results and feedback messages displayed on the user interface. Specifically, the terminal parses the JSON data and displays it in a format that is easy for the user to understand.

[0339] Specific examples

[0340] A user takes a picture of the newly installed antenna of the base station with their smartphone.

[0341] The user uses a communication application to send the image to the system.

[0342] The device receives the image, converts it into the appropriate format, and sends it to the server.

[0343] The server receives the images and analyzes them using an AI module.

[0344] The AI ​​module extracts characteristic information and compares it with a database.

[0345] The server identifies the matching component information, and in parallel, the emotion engine recognizes the user's emotion.

[0346] The server generates adjusted matching results and emotional feedback and sends them to the device.

[0347] The device will display the results, along with a message such as "This is an antenna with model number XX" and "Thank you for your hard work, it's almost done."

[0348] (Application example 2)

[0349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] Conventional parts identification work in factories often relies on the experience and knowledge of the worker, making it difficult for new employees or newly transferred workers to quickly and accurately identify parts. Furthermore, there was no mechanism to reduce stress and anxiety during work, leading to calls for improvements to the work environment. To solve these issues, a system is needed that automates the parts identification process and improves work efficiency by providing feedback on the user's emotions.

[0351] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving an image taken by a user, means for analyzing the image and extracting feature information, means for comparing the extracted feature information with information in a database, means for analyzing the user's emotions using an emotion engine, and means for generating a feedback message based on the analyzed emotion information. This allows factory workers to easily identify components, and by receiving feedback that takes into account their emotions during work, it is possible to reduce stress and improve work efficiency.

[0352] "User" refers to the factory worker who takes and sends images to the system for part identification and emotion recognition.

[0353] "Means for receiving images" refers to a communication function for allowing the server to receive images taken by the user.

[0354] The "means for analyzing an image and extracting characteristic information" refers to an image analysis function for extracting characteristic information such as the shape, text, and color of a component from a received image.

[0355] "Database" refers to data storage for storing component information and feature data.

[0356] "Means for matching" refers to a computer program for comparing extracted characteristic information with information in a database and identifying matching components.

[0357] "User's device" refers to the device used to receive and display matching results and feedback messages, specifically a smartphone, tablet, or PC.

[0358] An "emotion engine" refers to an algorithm or software that analyzes facial images and voice data taken by a user to generate emotional information.

[0359] "Emotion information" is data that indicates the user's emotional state, and includes emotions such as stress, anxiety, and joy.

[0360] A "feedback message" refers to a response message provided to a user based on emotional information.

[0361] "System" refers to the collection of devices and software that integrates and operates all of these means.

[0362] The present invention is a system for improving the efficiency of component identification work in a factory and for improving the work environment by providing feedback on the emotions of workers. Specific embodiments for carrying out the invention are described below.

[0363] System Overview

[0364] The system receives images taken by the user, analyzes them to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback based on the results. This allows workers to easily identify parts and receive feedback based on their emotions.

[0365] Hardware and software used

[0366] Hardware

[0367] Smartphone terminal: Used by factory workers to take and send images of parts.

[0368] Robot: Used to move autonomously within the factory and take images of components.

[0369] Server: Performs image analysis, database matching, emotion recognition, and feedback message generation.

[0370] software

[0371] Image analysis: Extracting feature information from images using TensorFlow and OpenCV.

[0372] Database: MySQL is used to store and manage component information.

[0373] Emotion Recognition: Analyze user emotions using Azure Cognitive Services and Google Cloud Natural Language API.

[0374] Communication applications: Use common communication applications (e.g., LINE or WhatsApp) to send images.

[0375] System operation explanation

[0376] Taking and sending images

[0377] The user takes a picture of the component using a smartphone or a camera mounted on the robot and sends it to the server via a communication application.

[0378] Image analysis and feature information extraction

[0379] The server analyzes the received images using TensorFlow and OpenCV to extract feature information such as shape, text, and color, allowing it to compare detailed information about the identified components with a database.

[0380] Emotion Recognition and Feedback

[0381] In parallel, the server analyzes the user's facial image and voice data using Azure Cognitive Services and Google Cloud Natural Language API to generate emotional information, and then generates a feedback message based on the results and sends it to the user's device.

[0382] Specific examples

[0383] For example, suppose a user takes a picture of a component (e.g., a screw) in a factory using their smartphone. The image is sent to a server via a communication application (e.g., LINE or WhatsApp). The server analyzes the image, extracts the component's characteristics, and compares them with a database. The emotion engine then analyzes the user's emotions and generates a feedback message such as "This screw is model number XX" or "Thank you for your hard work. The work is going well," which is then sent to the user's device.

[0384] Examples of prompt statements

[0385] image_path = 'path_to_image'

[0386] user_data = 'Information including voice data and user facial images'

[0387] result = main(image_path, user_data)

[0388] print(json.dumps(result, indent=4, ensure_ascii=False))

[0389] In this way, the present invention allows factory workers to quickly and accurately identify parts and contributes to improving the work environment by providing feedback on their emotions while working.

[0390] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0391] Step 1:

[0392] The user takes an image of the part. Using a smartphone or a camera mounted on a robot, the image of the part is captured. This generates image data, which then becomes the input for processing in the next step.

[0393] Step 2:

[0394] The user sends the image they have taken to their device via a communication application, such as LINE or WhatsApp. During this process, the image data is sent from the smartphone to the server.

[0395] Step 3:

[0396] The server temporarily stores the images received from the user. The image data is converted into an appropriate format (JPEG, PNG, etc.) and stored in the internal memory. This stored image data is used in the next analysis step.

[0397] Step 4:

[0398] The server analyzes the stored image data using TensorFlow and OpenCV. Image analysis extracts feature information such as shape, text, and color. This feature information becomes input data for matching with the database.

[0399] Step 5:

[0400] The server compares the extracted feature information with the component information in the database, which stores information such as component model numbers and names. As a result of the comparison, the component information that best matches the information is identified.

[0401] Step 6:

[0402] The server simultaneously collects the user's emotional information and analyzes it using Azure Cognitive Services and Google Cloud Natural Language API. Facial images and voice data are input as the user's emotional data, and this emotional data is analyzed to output emotional information such as stress, anxiety, and joy.

[0403] Step 7:

[0404] The server generates a feedback message based on the analysis results of the emotion engine. An appropriate feedback message is generated based on the component information of the collation result and the user's emotion information. This feedback message is then sent to the user terminal.

[0405] Step 8:

[0406] The terminal displays the matching results and feedback messages received from the server to the user. The user can check information on the terminal, such as "This screw is model number XX" or "Thank you for your hard work, the work is going well." This improves work efficiency and the user experience.

[0407] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0408] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0409] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0410] [Second embodiment]

[0411] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0412] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0413] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0414] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0415] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0416] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0417] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0418] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0419] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0420] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0421] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0422] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0423] This invention is a system for quickly and accurately identifying materials in base station construction work, allowing even new employees and transferred employees to contribute immediately to the work.

[0424] System Overview

[0425] The system receives images taken by the user, analyzes them to extract characteristic information, compares them with a database, and returns the results to the user, allowing the user to easily identify components.

[0426] Program processing flow

[0427] 1. Taking and sending images

[0428] Users take pictures of components (e.g., antennas, radios, and concrete poles) at base station sites using their smartphone cameras, and then use communication applications (e.g., LINE, WhatsApp) to send the images to the system.

[0429] 2. Receiving and temporarily storing images

[0430] The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this point, the device converts the image into the appropriate format (e.g., JPEG, PNG) and prepares it for transmission to the server.

[0431] 3. Image submission and analysis

[0432] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0433] 4. Matching feature information

[0434] The server compares the feature information extracted by the AI ​​module with the component information in the database, retrieves detailed information such as the product name and model number of the relevant component from the database, and selects the result with the best match.

[0435] 5. Generating and sending matching results

[0436] The server generates a match result to send to the user's device, encodes it as JSON data, and sends this data to the user's device as an HTTP response.

[0437] 6. Display of matching results

[0438] The device decodes the response received from the server and displays the matching result in a format that is easy for the user to understand. The user can then confirm the recognition result (e.g., "This is an antenna with model number XX") on the device.

[0439] Specific examples

[0440] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[0441] 2. The user sends the image to the system via a communication application.

[0442] 3. The device receives the image, converts it into the appropriate format, and sends it to the server.

[0443] 4. The server receives the image and passes it to the AI ​​module for analysis.

[0444] 5. The AI ​​module extracts the antenna's characteristic information and compares it with a database.

[0445] 6. The server identifies the product name and model number of the matching antenna and generates a match.

[0446] 7. The server sends the matching results to the user's device.

[0447] 8. The device displays the matching results to the user and notifies them that the antenna is model number XX.

[0448] Through this process, the system enables users to quickly identify components on-site, improving the efficiency and accuracy of on-site operations.

[0449] The processing flow will be explained below.

[0450] Step 1:

[0451] The user takes pictures of components (e.g., antennas, radios, and concrete poles) at the base station site using their smartphone camera.

[0452] Step 2:

[0453] The user opens a communication application (e.g., LINE, WhatsApp), selects the captured image, and presses the "Send" button to send it to the system.

[0454] Step 3:

[0455] The device receives the image via the communication application, temporarily stores the image data in its internal memory, and converts the image to an appropriate format (e.g., JPEG, PNG).

[0456] Step 4:

[0457] The device generates an HTTP POST request to send the saved image data to the server, and sends it to the server's image reception endpoint.

[0458] Step 5:

[0459] The server receives the HTTP request, extracts the image data from the request body, and temporarily stores it.

[0460] Step 6:

[0461] The server passes the stored image to the AI ​​module and begins image analysis.

[0462] Step 7:

[0463] The AI ​​module (in the server) analyzes the image and extracts feature information such as shape, text, and color.

[0464] Step 8:

[0465] Based on the feature information extracted by the AI ​​module, the server generates an SQL query to match it with component information in the database.

[0466] Step 9:

[0467] The server runs an SQL query against the database to obtain detailed information such as the product name and model number of the relevant part.

[0468] Step 10:

[0469] The server selects the best match, encodes the recognition result in JSON format, stores it in an HTTP response, and sends it to the user's device.

[0470] Step 11:

[0471] The device receives the response from the server and decodes the JSON data.

[0472] Step 12:

[0473] The device updates the GUI to display the recognition result (e.g., "This is an antenna with model number XX") to the user.

[0474] Step 13:

[0475] The user checks the recognition results displayed on the terminal and, if necessary, proceeds with further investigation or response.

[0476] Example 1

[0477] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0478] Identifying materials in base station construction work requires engineers with specialized knowledge, and it is difficult for new employees or transferred employees to quickly and accurately identify materials on-site. This can reduce work efficiency and affect accuracy. Furthermore, there is a risk of frequent mistakes and rework on-site, so a solution to these issues is needed.

[0479] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0480] In this invention, the server includes a device that receives images taken by a user, an analysis device that analyzes the images and extracts feature information, a comparison device that compares the extracted feature information with information in a database, and a communication device that transmits the comparison results to the user's device, thereby enabling the user to quickly and accurately identify components based on images taken at the site.

[0481] A "user" is a person who uses the system to identify components.

[0482] An "image" is a file that represents visual information of an object photographed by a user.

[0483] A "device" is a device for receiving images taken by a user.

[0484] An "analysis device" is a device that analyzes a received image and extracts feature information.

[0485] "Feature information" is discernible data such as shape, text, and color extracted from an image.

[0486] A "database" is a digital storage device that stores information to be matched with characteristic information.

[0487] A "matching device" is a device that matches the extracted feature information with information in a database.

[0488] A "communication device" is a device for transmitting the collation results to the user's device.

[0489] "Communication software" is a program for transmitting images taken by the user.

[0490] The "display device" is a device for displaying detailed information about a component based on its characteristic information.

[0491] The "system" is a set of devices and software that identifies components based on images taken by the user.

[0492] This invention relates to a system for quickly and accurately identifying components in base station construction work. The system analyzes images taken by users on-site, compares them with a database, and provides information on the relevant components. The specific system configuration, the hardware and software used, and the data processing method are described in detail below.

[0493] System configuration

[0494] The system mainly consists of the following components:

[0495] 1. Your device

[0496] A mobile device such as a smartphone or tablet is used, which has a camera and communication software (e.g., LINE, WhatsApp) installed.

[0497] 2. Analysis device

[0498] It uses an AI module located on the server, specifically consisting of hardware and software (e.g., TensorFlow, PyTorch) for executing image recognition algorithms.

[0499] 3. Database

[0500] A central database that stores component information, including detailed information such as component product name, model number, shape, and color.

[0501] 4. Communications Equipment

[0502] It includes the network infrastructure (e.g. HTTP server, RESTful API) for communication between the server and the user's device.

[0503] Specific program processing and usage

[0504] 1. Taking and sending images

[0505] The user takes an image of the component at the base station site using the smartphone camera, then launches the communication software on the smartphone and sends the image to the system.

[0506] 2. Receiving and temporarily storing images

[0507] The device receives the image via the communication software, temporarily stores it in its internal memory, and converts it into the appropriate format (e.g., JPEG, PNG) as needed.

[0508] 3. Image submission and analysis

[0509] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0510] 4. Matching feature information

[0511] The server compares the feature information extracted by the AI ​​module with the component information in the database, selects the most matching result, and obtains detailed information.

[0512] 5. Generating and sending matching results

[0513] The server generates a match result to send to the user's device, encodes it as JSON data, and sends it to the user's device as an HTTP response.

[0514] 6. Display of matching results

[0515] The device decodes the response received from the server and displays the matching results in a format that is easy for the user to understand, such as a message like "This is an antenna with model number XX."

[0516] Specific examples

[0517] For example:

[0518] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[0519] 2. The user selects an image in the LINE app and sends it to the system.

[0520] 3. The device receives the image, converts it to JPEG format, and saves it to its internal memory.

[0521] 4. The device sends the saved image to the server.

[0522] 5. The server receives the image and passes it to the AI ​​module for analysis.

[0523] 6. The AI ​​module extracts features such as antenna shape, text, and color and compares them with a database.

[0524] 7. The server identifies the product name and model number of the best matching antenna and generates the results in JSON format.

[0525] 8. The server sends the generated results to the user's device.

[0526] 9. The device analyzes the results received and displays to the user, "This is an antenna with model number XX."

[0527] Example prompts for generative AI models

[0528] Possible prompts include:

[0529] "You are developing a system to identify images of base station antennas. Follow these steps to describe a program that will analyze the image, match it against a database, and return the results."

[0530] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0531] Program processing steps

[0532] Step 1: User takes and submits image

[0533] The user takes an image of the components (e.g., antenna, radio, concrete pole) at the base station site using the smartphone camera. When taking the photo, the camera is adjusted so that the components are clearly visible. After taking the photo, the user launches a communication application on the smartphone (e.g., LINE, WhatsApp), selects the image, and sends it to the system.

[0534] Input: Image taken by the camera

[0535] Output: Image data sent to the system via a communication application

[0536] Step 2: Receiving and temporarily saving images on the device

[0537] The device receives images sent by the user via a communication application. The received image data is temporarily stored in internal memory. When saving, the device checks whether the image is in an appropriate format (e.g., JPEG, PNG), and if it is not, converts it to an appropriate format. It also checks that the image resolution is within a range suitable for analysis.

[0538] Input: Image data sent from a communication application

[0539] Output: Image data in the appropriate format, stored in the internal memory

[0540] Step 3: Sending images from the device to the server and analyzing them

[0541] The device generates a request to send pre-prepared image data to the server. This request also includes metadata (e.g., date and time of transmission, user ID). Once the request is sent, the server receives the request and saves the image data in a folder for internal processing. The server then passes the saved image data to the AI ​​module, which starts the image analysis process. The AI ​​module first pre-processes the image, removing noise and filtering, and then extracts feature information such as shape, text, and color.

[0542] Input: Image data stored in the internal memory

[0543] Output: Feature information extracted by the AI ​​module

[0544] Step 4: Server checks the feature information

[0545] The server compares the feature information extracted by the AI ​​module with the component information in the database. It uses a high-speed search algorithm (e.g., KD-Tree, Locality-Sensitive Hashing) to find the best match. It retrieves the product name, model number, and related details of the component from the database. If there are multiple candidates, it selects the best one.

[0546] Input: Feature information provided by the AI ​​module

[0547] Output: Component information extracted from the database (product name, model number, detailed information)

[0548] Step 5: Server generates and sends match results

[0549] The server generates a matching result to send to the user's device. This matching result is encoded as JSON data and includes necessary information (e.g., component product name, model number, and links to related documents). The server generates an HTTP response, embeds the JSON data in it, and sends it to the user's device.

[0550] Input: Material information extracted from the database

[0551] Output: Match results encoded in JSON format

[0552] Step 6: Display the match results on your device

[0553] The device analyzes the HTTP response received from the server and obtains the decoded JSON data. The device displays this data in a format that is easy for the user to understand. Specifically, the device displays a message such as "This is an antenna with model number XX" on the user screen, along with a link to more detailed information. The user can confirm this and check the detailed information as needed.

[0554] Input: JSON data of the match result sent from the server

[0555] Output: Material information displayed to the user

[0556] (Application example 1)

[0557] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] In factories, the task of identifying parts is typically manual and often time-consuming and labor-intensive. Furthermore, it is difficult for inexperienced workers to accurately grasp the information on identified parts, which can lead to problems due to misidentification. This can reduce work efficiency and negatively impact the productivity of the entire process. To solve these problems, a system that can quickly and effectively identify and manage parts is needed.

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0560] In this invention, the server includes means for receiving images taken by a user, means for analyzing the images and extracting feature information, means for comparing the extracted feature information with information in a database, means for transmitting the comparison result to the user's terminal, means for the robot to take an image of a component, means for saving the image data in a built-in computer and transmitting the image data to an analysis module after converting its format, means for extracting feature information such as shape, text, and color using the image analysis module, and means for reporting component information to a management system using the extracted feature information. This enables the robot to automatically identify components in a factory and report information to the management system in real time.

[0561] "User" refers to a person who uses this system to take images and transmit data.

[0562] "Image" refers to still image data captured by a camera.

[0563] "Means for receiving" refers to devices or software that have the function of receiving and storing captured image data.

[0564] "Means for analyzing" refers to algorithms or software for extracting feature information from received image data.

[0565] "Feature information" refers to data such as shape, text, and color obtained through image analysis.

[0566] "Database" refers to a collection of information that stores detailed information about components.

[0567] "Means for matching" refers to a device or software that has the function of comparing extracted feature information with information in a database to find a match.

[0568] The "matching result" refers to the specific information of the component obtained by the matching means.

[0569] "Terminal" refers to an electronic device used by a user, such as a smartphone or computer.

[0570] "Robot" refers to a self-propelled mechanical device used to identify and manage parts within a factory.

[0571] "On-board computer" refers to a data processing device installed in a robot.

[0572] "Image analysis module" refers to a program or software for analyzing image data and extracting feature information.

[0573] "Management system" refers to a system for managing and displaying information about components within a factory.

[0574] This invention relates to a system in which a robot automatically identifies parts in a factory and reports the information to a management system in real time. This system implements a series of processes: receiving images taken by a user, analyzing them, extracting characteristic information, comparing them with a database, and sending the results. Furthermore, the system also includes functions for the robot to take images of parts, save the data in its built-in computer, send it to an analysis module, extract characteristic information, and report the part information to the management system.

[0575] The server has a means for receiving images taken by the user. For example, when a user takes an image with a smartphone and sends it via a communication application, the server receives the image and temporarily stores it.

[0576] Next, the server has a means to analyze the image and extract feature information. Here, an AI-based image analysis module is used, for example, the OpenCV library. This module extracts feature information such as shape, text, and color from the image data.

[0577] The server has a means for comparing the extracted feature information with information in a database. The database stores component information, and the server compares it with the information that best matches it.

[0578] The server has a means to send the matching results to the user's device. This means sends JSON format data to the user's device using an HTTP request. The user can check the results on their smartphone or other device.

[0579] The robot also has a means of taking images of the parts as it moves. The images are stored in the robot's onboard computer and converted to the appropriate format as needed. This computer then sends the image data to the analysis module, which then begins the image analysis.

[0580] The onboard computer uses an image analysis module to extract feature information such as shape, text, and color. Again, the OpenCV library can be used. The extracted feature information is reported to the management system.

[0581] For example, to identify new machine parts in a factory, a robot patrols a parts warehouse and takes pictures. The images are sent to a system, where they are analyzed and the results are reported to a management system. Workers can use this information to work more efficiently.

[0582] An example of a prompt for the generative AI model is, "Extract the feature information of the component from the image and compare it with the database. Identify the product name and model number of the corresponding component and display them to the user." Using this prompt, the AI ​​module can efficiently analyze the feature information and identify the component.

[0583] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0584] Step 1:

[0585] The user takes an image of a component using a smartphone or robot camera. The input is real-time image data obtained by the camera, and the output is an image file (e.g., JPEG, PNG).

[0586] Step 2:

[0587] The image taken by the user or robot is sent to the server via a communication application. Specifically, the user sends the image from their smartphone using a communication application (e.g., LINE, WhatsApp), and the robot similarly sends it to the server using its built-in communication module. The purpose of this step is for the image data to reach the server. The input is the captured image file, and the output is the sent image data.

[0588] Step 3:

[0589] The server temporarily stores the received image data. This storage process includes converting the image data to the appropriate file format (e.g., converting JPEG to PNG). The input is the raw image data received by the server, and the output is image data that has been appropriately saved in the temporary storage location, such as memory or storage.

[0590] Step 4:

[0591] The server passes the saved image data to the analysis module and begins image analysis. The analysis uses libraries such as OpenCV to extract feature information such as shape, text, and color from the image. The input is the temporarily saved image data, and the output is the extracted feature information. Specific operations include image filtering, text recognition, and shape analysis using OpenCV.

[0592] Step 5:

[0593] The server compares the extracted feature information with information in the database. Here, a comparison is made with pre-registered component information to identify the component information that best matches. The input is the extracted feature information and component information in the database, and the output is the specific information of the component that best matches (e.g., product name, model number).

[0594] Step 6:

[0595] The server generates matching results and sends them to the user's device. The generated matching results are encoded in a format such as JSON and sent to the user's device as an HTTP request. The input is information about the identified components, and the output is the encoded matching result data.

[0596] Step 7:

[0597] The terminal decodes the matching results received from the server and displays them to the user. The user can check the results on their smartphone or the management system. The input is the matching result data sent from the server, and the output is information displayed in a format that is easy for the user to understand. Specific operations include parsing JSON data and displaying the information on the UI.

[0598] Step 8:

[0599] The robot then patrols the parts in the factory, taking new images of the parts and sending them to the system. In this step, the robot's built-in computer stores the image data, passes it to an analysis module to extract feature information, and reports it to the management system. The input is the new image data taken by the robot, and the output is the part information reported to the management system.

[0600] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0601] This invention is a system for quickly and accurately identifying components in base station construction work, and aims to improve the user experience by recognizing the user's emotions and providing feedback based on those emotions. This allows even new employees and transferred employees to contribute to the work as an immediate asset.

[0602] System Overview

[0603] The system receives images taken by the user, analyzes them to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback on the results, achieving a series of steps. This allows the user to easily identify components, and the system recognizes and provides feedback on the user's emotions during the process, reducing stress for the user.

[0604] Program processing flow

[0605] 1. Taking and sending images

[0606] Users take pictures of components (e.g., antennas, radios, and concrete poles) at base station sites using their smartphone cameras, and then use communication applications (e.g., LINE or WhatsApp) to send the images to the system.

[0607] 2. Receiving and temporarily storing images

[0608] The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this point, the device converts the image into the appropriate format (e.g., JPEG, PNG) and prepares it for transmission to the server.

[0609] 3. Image submission and analysis

[0610] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0611] 4. Matching feature information

[0612] The server compares the feature information extracted by the AI ​​module with the component information in the database, retrieves detailed information such as the product name and model number of the relevant component from the database, and selects the result with the best match.

[0613] 5. Emotion Recognition and Feedback

[0614] In parallel with image matching, the server runs an emotion engine that extracts emotional information from the user's facial image, voice, etc. The emotion engine recognizes the user's emotions (e.g., stress, anxiety, joy, etc.) and generates analysis results.

[0615] 6. Generating and transmitting emotion-based matching results

[0616] The server generates a match result adjusted based on the analysis results of the emotion engine and sends it to the user's device, which also includes a feedback message according to the user's emotion.

[0617] 7. Displaying matching results and feedback

[0618] The device decodes the response received from the server and displays the matching result and feedback in a format that is easy for the user to understand. The user can see the recognition result (e.g., "This is an antenna with model number XX") on the device along with a feedback message corresponding to the user's emotion.

[0619] Specific examples

[0620] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[0621] 2. The user sends the image to the system via a communication application.

[0622] 3. The device receives the image, converts it into the appropriate format, and sends it to the server.

[0623] 4. The server receives the image and passes it to the AI ​​module for analysis.

[0624] 5. The AI ​​module extracts the antenna's characteristic information and compares it with a database.

[0625] 6. The server identifies the product name and model number of the matching antenna, and in parallel the emotion engine recognizes the user's emotion.

[0626] 7. The server generates feedback based on the matching results and the user's emotions and sends it to the user's device as JSON format data.

[0627] 8. The device displays the matching results and feedback to the user, notifying them with a message such as "This is an antenna with model number XX" and "Good job, almost done."

[0628] Through this process, the system can help users quickly identify parts on-site, while also recognizing users' emotions and providing appropriate feedback to improve the user experience.

[0629] The processing flow will be explained below.

[0630] Step 1:

[0631] The user takes pictures of components (e.g., antennas, radios, and concrete poles) at the base station site using the smartphone camera.

[0632] Step 2:

[0633] The user opens a communication application (e.g., LINE, WhatsApp), selects the captured image, and presses the "Send" button to send it to the system.

[0634] Step 3:

[0635] The device receives the image via the communication application, temporarily stores the image data in its internal memory, and converts the image to an appropriate format (e.g., JPEG, PNG).

[0636] Step 4:

[0637] The device generates an HTTP POST request to send the saved image data to the server, and sends it to the server's image reception endpoint.

[0638] Step 5:

[0639] The server receives the HTTP request, extracts the image data from the request body, and temporarily stores it.

[0640] Step 6:

[0641] The server passes the stored image to the AI ​​module and begins image analysis.

[0642] Step 7:

[0643] The AI ​​module (in the server) analyzes the image and extracts feature information such as shape, text, and color.

[0644] Step 8:

[0645] Based on the feature information extracted by the AI ​​module, the server generates an SQL query to match it with component information in the database.

[0646] Step 9:

[0647] The server runs an SQL query against the database to obtain detailed information such as the product name and model number of the relevant part.

[0648] Step 10:

[0649] The server generates the matching results and simultaneously runs an emotion engine that extracts emotion information from the user's facial image, voice, etc.

[0650] Step 11:

[0651] The emotion engine (in the server) recognizes the user's emotions (e.g., stress, anxiety, joy, etc.) and generates the analysis results.

[0652] Step 12:

[0653] The server generates an adjusted matching result based on the analysis results of the emotion engine and sends it to the user's device, which also includes a feedback message according to the user's emotion.

[0654] Step 13:

[0655] The device receives the response from the server and decodes the JSON data.

[0656] Step 14:

[0657] The terminal displays the recognition result (e.g., "This is an antenna with model number XX") to the user, as well as a feedback message according to the user's emotion (e.g., "Good job, you're almost done").

[0658] Step 15:

[0659] The user checks the recognition results and feedback messages displayed on the terminal, and if necessary, proceeds with further investigation or response.

[0660] This specific step allows the system to help users quickly identify components in the field, and also recognizes the user's emotions and provides appropriate feedback.

[0661] Example 2

[0662] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0663] In base station construction work, there is a need for a method that allows even new employees or newly transferred employees to quickly and accurately identify components. There is also a need for a feedback function to reduce the stress and anxiety of users who work long hours. There has not been a system that meets these needs, so there is a need to solve these issues.

[0664] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0665] In this invention, the server includes means for receiving an image taken by a user, means for analyzing the image and extracting feature information, means for comparing the extracted feature information with information in a database, means for recognizing the user's emotion, means for generating feedback based on the emotion recognition result, and means for transmitting the generated feedback to the user's terminal. This allows the user to easily identify components and receive appropriate feedback regarding changes in emotion during the process, thereby improving work efficiency and the user experience.

[0666] The "means for receiving images taken by a user" refers to a device or method for acquiring image data taken by a user with a photographing device such as a smartphone through communication.

[0667] "Means for analyzing images and extracting feature information" refers to a device or method that analyzes captured image data using machine learning algorithms or image processing techniques to extract features such as shape, text, and color within the image.

[0668] The "means for matching with information in a database" refers to a device or method that searches for items that match information in a pre-registered database based on the extracted characteristic information and generates a matching result.

[0669] "Means for recognizing user emotions" refers to a device or method for analyzing a user's facial expressions and tone of voice from image or audio data, etc., to detect the user's emotional state (e.g., stress, anxiety, joy, etc.).

[0670] The "means for generating feedback based on emotion recognition results" refers to a device or method for creating an appropriate feedback message according to the recognized emotional state of the user and notifying the user of the appropriate feedback message.

[0671] The "means for transmitting the generated feedback to the user's terminal" refers to a device or method for transmitting the generated feedback message from the server to the user's terminal such as a smartphone or tablet via data communication.

[0672] This invention is a system for quickly and accurately identifying components in base station construction work. It aims to improve the user experience by recognizing the user's emotions and providing feedback based on those emotions. This allows even new employees and newly transferred employees to contribute immediately to the work. The system receives images taken by the user, analyzes the images to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback based on the results.

[0673] The specific configuration of the system is as follows.

[0674] 1. Receiving images

[0675] The user uses the smartphone camera to take an image of the component at the base station site. The captured image is then sent to the system using a communication application (e.g., LINE or WhatsApp). The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this time, the device converts the image into an appropriate format (e.g., JPEG or PNG).

[0676] 2. Sending images and preparing for analysis

[0677] The device generates a request to send the stored image data to the server and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. This AI module performs image analysis using a deep learning framework (e.g., TensorFlow, PyTorch).

[0678] 3. Extraction of feature information

[0679] The server uses an AI module to extract feature information such as shape, text, and color from the image, which is then used for database matching.

[0680] 4. Database Matching

[0681] The server compares the extracted feature information with the registered information in the database. The database is built using a common RDBMS (e.g., MySQL or PostgreSQL) and stores detailed information such as the product name and model number of the component. The server selects the most matching result and temporarily stores it.

[0682] 5. Performing Emotion Recognition

[0683] In parallel, the emotion engine extracts emotional information from the user's facial image, voice, and other data. The emotion engine performs emotion analysis using, for example, OpenCV or a natural language processing (NLP) model. The recognized emotional data (e.g., stress level, anxiety level, happiness level) is saved.

[0684] 6. Producing adjusted matching results

[0685] The server adds a feedback message to the matching result based on the emotion recognition results. For example, if the user is feeling stressed, it generates a message such as "Good job, you're almost done." This data is converted into JSON format and sent to the user's device.

[0686] 7. Viewing Results and Feedback

[0687] The device receives the response from the server and displays the analysis results and feedback on the user interface. The user can check the recognition result (e.g., "This is an antenna with model number XX") and the emotional feedback message on the device.

[0688] Specific examples

[0689] A user takes a picture of the newly installed antenna of the base station with their smartphone.

[0690] The user uses a communication application to send the image to the system.

[0691] The device receives the image, converts it into the appropriate format, and sends it to the server.

[0692] The server receives the images and analyzes them using an AI module.

[0693] The AI ​​module extracts characteristic information and compares it with a database.

[0694] The server identifies the matching component information, and in parallel, the emotion engine recognizes the user's emotion.

[0695] The server generates adjusted matching results and emotional feedback and sends them to the device.

[0696] The device will display the results, along with a message such as "This is an antenna with model number XX" and "Thank you for your hard work, it's almost done."

[0697] Prompt Sentence Examples

[0698] "I would like to develop a system that analyzes and identifies images of components taken at the base station's work site and returns the results. It also needs a function that recognizes the user's emotions and provides feedback. Please explain in detail the program flow for such a system."

[0699] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0700] Step 1:

[0701] The user uses the smartphone camera to take pictures of components (e.g., antennas, radios, and concrete poles) at the base station site. The input is the image taken by the user with the camera, and the output is an image file saved on the smartphone. Specifically, the user launches the smartphone's camera app, frames the component, and presses the capture button.

[0702] Step 2:

[0703] The user sends the captured image to the system using a communication application (e.g., LINE or WhatsApp). The input is an image file stored on the smartphone, and the output is image data sent through the communication application. Specifically, the user opens the communication application, displays the chat screen, attaches the image, and presses the send button.

[0704] Step 3:

[0705] The device receives images via a communication application and temporarily stores the image data in its internal memory. The input is the image data received from the communication application, and the output is an image file stored in a temporary storage folder. Specifically, the device receives an application notification, downloads the image data, and stores it in a temporary storage folder (e.g., / tmp / images).

[0706] Step 4:

[0707] The device converts the received image into an appropriate format (e.g., JPEG, PNG) and generates a request to send to the server. The input is the image file saved in the temporary storage folder, and the output is the image data sent to the server as an HTTP POST request. Specifically, the device checks the image format, performs conversion processing if necessary, and generates an HTTP POST request.

[0708] Step 5:

[0709] The server passes the received image data to the AI ​​module and begins image analysis. The input is the image data received as an HTTP POST request, and the output is the image data waiting to be analyzed. Specifically, the server obtains the image data via the file reception API and passes it to the AI ​​module.

[0710] Step 6:

[0711] The server uses an AI module to extract feature information such as shape, text, and color from images. The input is the image data passed to the AI ​​module, and the output is the extracted feature information. Specifically, the server runs an AI module (e.g., TensorFlow, PyTorch) and applies an image analysis algorithm to extract feature information.

[0712] Step 7:

[0713] The server compares the extracted feature information with the component information in the database. The input is the extracted feature information, and the output is the component information as a comparison result. Specifically, the server generates a database query and searches a database such as MySQL or PostgreSQL to obtain the most matching component information.

[0714] Step 8:

[0715] The server also uses an emotion engine to recognize the user's emotions. The input is the user's facial image and voice data, and the output is the recognized emotion data. Specifically, the server runs the emotion engine (e.g., OpenCV, NLP model) to extract emotion information from the facial image and voice.

[0716] Step 9:

[0717] The server generates an adjusted matching result and a feedback message based on the emotion recognition result. The input is the component information as the matching result and the recognized emotion data, and the output is the adjusted matching result and a feedback message. Specifically, the server creates a feedback message based on the emotion data and converts it into JSON format.

[0718] Step 10:

[0719] The server sends the generated feedback message to the user's device. The input is the matching result and feedback message converted to JSON format, and the output is the data sent to the user's device as an HTTP response. Specifically, the server sends the JSON data through an API endpoint.

[0720] Step 11:

[0721] The terminal receives the response from the server and displays the analysis results and feedback on the user interface. The input is the JSON data received from the server, and the output is the recognition results and feedback messages displayed on the user interface. Specifically, the terminal parses the JSON data and displays it in a format that is easy for the user to understand.

[0722] Specific examples

[0723] A user takes a picture of the newly installed antenna of the base station with their smartphone.

[0724] The user uses a communication application to send the image to the system.

[0725] The device receives the image, converts it into the appropriate format, and sends it to the server.

[0726] The server receives the images and analyzes them using an AI module.

[0727] The AI ​​module extracts characteristic information and compares it with a database.

[0728] The server identifies the matching component information, and in parallel, the emotion engine recognizes the user's emotion.

[0729] The server generates adjusted matching results and emotional feedback and sends them to the device.

[0730] The device will display the results, along with a message such as "This is an antenna with model number XX" and "Thank you for your hard work, it's almost done."

[0731] (Application example 2)

[0732] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0733] Conventional parts identification work in factories often relies on the experience and knowledge of the worker, making it difficult for new employees or newly transferred workers to quickly and accurately identify parts. Furthermore, there was no mechanism to reduce stress and anxiety during work, leading to calls for improvements to the work environment. To solve these issues, a system is needed that automates the parts identification process and improves work efficiency by providing feedback on the user's emotions.

[0734] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving an image taken by a user, means for analyzing the image and extracting feature information, means for comparing the extracted feature information with information in a database, means for analyzing the user's emotions using an emotion engine, and means for generating a feedback message based on the analyzed emotion information. This allows factory workers to easily identify components, and by receiving feedback that takes into account their emotions during work, it is possible to reduce stress and improve work efficiency.

[0735] "User" refers to the factory worker who takes and sends images to the system for part identification and emotion recognition.

[0736] "Means for receiving images" refers to a communication function for allowing the server to receive images taken by the user.

[0737] The "means for analyzing an image and extracting characteristic information" refers to an image analysis function for extracting characteristic information such as the shape, text, and color of a component from a received image.

[0738] "Database" refers to data storage for storing component information and feature data.

[0739] "Means for matching" refers to a computer program for comparing extracted characteristic information with information in a database and identifying matching components.

[0740] "User's device" refers to the device used to receive and display matching results and feedback messages, specifically a smartphone, tablet, or PC.

[0741] An "emotion engine" refers to an algorithm or software that analyzes facial images and voice data taken by a user to generate emotional information.

[0742] "Emotion information" is data that indicates the user's emotional state, and includes emotions such as stress, anxiety, and joy.

[0743] A "feedback message" refers to a response message provided to a user based on emotional information.

[0744] "System" refers to the collection of devices and software that integrates and operates all of these means.

[0745] The present invention is a system for improving the efficiency of component identification work in a factory and for improving the work environment by providing feedback on the emotions of workers. Specific embodiments for carrying out the invention are described below.

[0746] System Overview

[0747] The system receives images taken by the user, analyzes them to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback based on the results. This allows workers to easily identify parts and receive feedback based on their emotions.

[0748] Hardware and software used

[0749] Hardware

[0750] Smartphone terminal: Used by factory workers to take and send images of parts.

[0751] Robot: Used to move autonomously within the factory and take images of components.

[0752] Server: Performs image analysis, database matching, emotion recognition, and feedback message generation.

[0753] software

[0754] Image analysis: Extracting feature information from images using TensorFlow and OpenCV.

[0755] Database: MySQL is used to store and manage component information.

[0756] Emotion Recognition: Analyze user emotions using Azure Cognitive Services and Google Cloud Natural Language API.

[0757] Communication applications: Use common communication applications (e.g., LINE or WhatsApp) to send images.

[0758] System operation explanation

[0759] Taking and sending images

[0760] The user takes a picture of the component using a smartphone or a camera mounted on the robot and sends it to the server via a communication application.

[0761] Image analysis and feature information extraction

[0762] The server analyzes the received images using TensorFlow and OpenCV to extract feature information such as shape, text, and color, allowing it to compare detailed information about the identified components with a database.

[0763] Emotion Recognition and Feedback

[0764] In parallel, the server analyzes the user's facial image and voice data using Azure Cognitive Services and Google Cloud Natural Language API to generate emotional information, and then generates a feedback message based on the results and sends it to the user's device.

[0765] Specific examples

[0766] For example, suppose a user takes a picture of a component (e.g., a screw) in a factory using their smartphone. The image is sent to a server via a communication application (e.g., LINE or WhatsApp). The server analyzes the image, extracts the component's characteristics, and compares them with a database. The emotion engine then analyzes the user's emotions and generates a feedback message such as "This screw is model number XX" or "Thank you for your hard work. The work is going well," which is then sent to the user's device.

[0767] Examples of prompt statements

[0768] image_path = 'path_to_image'

[0769] user_data = 'Information including voice data and user facial images'

[0770] result = main(image_path, user_data)

[0771] print(json.dumps(result, indent=4, ensure_ascii=False))

[0772] In this way, the present invention allows factory workers to quickly and accurately identify parts and contributes to improving the work environment by providing feedback on their emotions while working.

[0773] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0774] Step 1:

[0775] The user takes an image of the part. Using a smartphone or a camera mounted on a robot, the image of the part is captured. This generates image data, which then becomes the input for processing in the next step.

[0776] Step 2:

[0777] The user sends the image they have taken to their device via a communication application, such as LINE or WhatsApp. During this process, the image data is sent from the smartphone to the server.

[0778] Step 3:

[0779] The server temporarily stores the images received from the user. The image data is converted into an appropriate format (JPEG, PNG, etc.) and stored in the internal memory. This stored image data is used in the next analysis step.

[0780] Step 4:

[0781] The server analyzes the stored image data using TensorFlow and OpenCV. Image analysis extracts feature information such as shape, text, and color. This feature information becomes input data for matching with the database.

[0782] Step 5:

[0783] The server compares the extracted feature information with the component information in the database, which stores information such as component model numbers and names. As a result of the comparison, the component information that best matches the information is identified.

[0784] Step 6:

[0785] The server simultaneously collects the user's emotional information and analyzes it using Azure Cognitive Services and Google Cloud Natural Language API. Facial images and voice data are input as the user's emotional data, and this emotional data is analyzed to output emotional information such as stress, anxiety, and joy.

[0786] Step 7:

[0787] The server generates a feedback message based on the analysis results of the emotion engine. An appropriate feedback message is generated based on the component information of the collation result and the user's emotion information. This feedback message is then sent to the user terminal.

[0788] Step 8:

[0789] The terminal displays the matching results and feedback messages received from the server to the user. The user can check information on the terminal, such as "This screw is model number XX" or "Thank you for your hard work, the work is going well." This improves work efficiency and the user experience.

[0790] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0791] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0792] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0793] [Third embodiment]

[0794] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0795] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0796] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0797] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0798] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0799] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0800] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0801] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0802] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0803] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0804] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0805] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0806] This invention is a system for quickly and accurately identifying materials in base station construction work, allowing even new employees and transferred employees to contribute immediately to the work.

[0807] System Overview

[0808] The system receives images taken by the user, analyzes them to extract characteristic information, compares them with a database, and returns the results to the user, allowing the user to easily identify components.

[0809] Program processing flow

[0810] 1. Taking and sending images

[0811] Users take pictures of components (e.g., antennas, radios, and concrete poles) at base station sites using their smartphone cameras, and then use communication applications (e.g., LINE, WhatsApp) to send the images to the system.

[0812] 2. Receiving and temporarily storing images

[0813] The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this point, the device converts the image into the appropriate format (e.g., JPEG, PNG) and prepares it for transmission to the server.

[0814] 3. Image submission and analysis

[0815] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0816] 4. Matching feature information

[0817] The server compares the feature information extracted by the AI ​​module with the component information in the database, retrieves detailed information such as the product name and model number of the relevant component from the database, and selects the result with the best match.

[0818] 5. Generating and sending matching results

[0819] The server generates a match result to send to the user's device, encodes it as JSON data, and sends this data to the user's device as an HTTP response.

[0820] 6. Display of matching results

[0821] The device decodes the response received from the server and displays the matching result in a format that is easy for the user to understand. The user can then confirm the recognition result (e.g., "This is an antenna with model number XX") on the device.

[0822] Specific examples

[0823] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[0824] 2. The user sends the image to the system via a communication application.

[0825] 3. The device receives the image, converts it into the appropriate format, and sends it to the server.

[0826] 4. The server receives the image and passes it to the AI ​​module for analysis.

[0827] 5. The AI ​​module extracts the antenna's characteristic information and compares it with a database.

[0828] 6. The server identifies the product name and model number of the matching antenna and generates a match.

[0829] 7. The server sends the matching results to the user's device.

[0830] 8. The device displays the matching results to the user and notifies them that the antenna is model number XX.

[0831] Through this process, the system enables users to quickly identify components on-site, improving the efficiency and accuracy of on-site operations.

[0832] The processing flow will be explained below.

[0833] Step 1:

[0834] The user takes pictures of components (e.g., antennas, radios, and concrete poles) at the base station site using their smartphone camera.

[0835] Step 2:

[0836] The user opens a communication application (e.g., LINE, WhatsApp), selects the captured image, and presses the "Send" button to send it to the system.

[0837] Step 3:

[0838] The device receives the image via the communication application, temporarily stores the image data in its internal memory, and converts the image to an appropriate format (e.g., JPEG, PNG).

[0839] Step 4:

[0840] The device generates an HTTP POST request to send the saved image data to the server, and sends it to the server's image reception endpoint.

[0841] Step 5:

[0842] The server receives the HTTP request, extracts the image data from the request body, and temporarily stores it.

[0843] Step 6:

[0844] The server passes the stored image to the AI ​​module and begins image analysis.

[0845] Step 7:

[0846] The AI ​​module (in the server) analyzes the image and extracts feature information such as shape, text, and color.

[0847] Step 8:

[0848] Based on the feature information extracted by the AI ​​module, the server generates an SQL query to match it with component information in the database.

[0849] Step 9:

[0850] The server runs an SQL query against the database to obtain detailed information such as the product name and model number of the relevant part.

[0851] Step 10:

[0852] The server selects the best match, encodes the recognition result in JSON format, stores it in an HTTP response, and sends it to the user's device.

[0853] Step 11:

[0854] The device receives the response from the server and decodes the JSON data.

[0855] Step 12:

[0856] The device updates the GUI to display the recognition result (e.g., "This is an antenna with model number XX") to the user.

[0857] Step 13:

[0858] The user checks the recognition results displayed on the terminal and, if necessary, proceeds with further investigation or response.

[0859] Example 1

[0860] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0861] Identifying materials in base station construction work requires engineers with specialized knowledge, and it is difficult for new employees or transferred employees to quickly and accurately identify materials on-site. This can reduce work efficiency and affect accuracy. Furthermore, there is a risk of frequent mistakes and rework on-site, so a solution to these issues is needed.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0863] In this invention, the server includes a device that receives images taken by a user, an analysis device that analyzes the images and extracts feature information, a comparison device that compares the extracted feature information with information in a database, and a communication device that transmits the comparison results to the user's device, thereby enabling the user to quickly and accurately identify components based on images taken at the site.

[0864] A "user" is a person who uses the system to identify components.

[0865] An "image" is a file that represents visual information of an object photographed by a user.

[0866] A "device" is a device for receiving images taken by a user.

[0867] An "analysis device" is a device that analyzes a received image and extracts feature information.

[0868] "Feature information" is discernible data such as shape, text, and color extracted from an image.

[0869] A "database" is a digital storage device that stores information to be matched with characteristic information.

[0870] A "matching device" is a device that matches the extracted feature information with information in a database.

[0871] A "communication device" is a device for transmitting the collation results to the user's device.

[0872] "Communication software" is a program for transmitting images taken by the user.

[0873] The "display device" is a device for displaying detailed information about a component based on its characteristic information.

[0874] The "system" is a set of devices and software that identifies components based on images taken by the user.

[0875] This invention relates to a system for quickly and accurately identifying components in base station construction work. The system analyzes images taken by users on-site, compares them with a database, and provides information on the relevant components. The specific system configuration, the hardware and software used, and the data processing method are described in detail below.

[0876] System configuration

[0877] The system mainly consists of the following components:

[0878] 1. Your device

[0879] A mobile device such as a smartphone or tablet is used, which has a camera and communication software (e.g., LINE, WhatsApp) installed.

[0880] 2. Analysis device

[0881] It uses an AI module located on the server, specifically consisting of hardware and software (e.g., TensorFlow, PyTorch) for executing image recognition algorithms.

[0882] 3. Database

[0883] A central database that stores component information, including detailed information such as component product name, model number, shape, and color.

[0884] 4. Communications Equipment

[0885] It includes the network infrastructure (e.g. HTTP server, RESTful API) for communication between the server and the user's device.

[0886] Specific program processing and usage

[0887] 1. Taking and sending images

[0888] The user takes an image of the component at the base station site using the smartphone camera, then launches the communication software on the smartphone and sends the image to the system.

[0889] 2. Receiving and temporarily storing images

[0890] The device receives the image via the communication software, temporarily stores it in its internal memory, and converts it into the appropriate format (e.g., JPEG, PNG) as needed.

[0891] 3. Image submission and analysis

[0892] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0893] 4. Matching feature information

[0894] The server compares the feature information extracted by the AI ​​module with the component information in the database, selects the most matching result, and obtains detailed information.

[0895] 5. Generating and sending matching results

[0896] The server generates a match result to send to the user's device, encodes it as JSON data, and sends it to the user's device as an HTTP response.

[0897] 6. Display of matching results

[0898] The device decodes the response received from the server and displays the matching results in a format that is easy for the user to understand, such as a message like "This is an antenna with model number XX."

[0899] Specific examples

[0900] For example:

[0901] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[0902] 2. The user selects an image in the LINE app and sends it to the system.

[0903] 3. The device receives the image, converts it to JPEG format, and saves it to its internal memory.

[0904] 4. The device sends the saved image to the server.

[0905] 5. The server receives the image and passes it to the AI ​​module for analysis.

[0906] 6. The AI ​​module extracts features such as antenna shape, text, and color and compares them with a database.

[0907] 7. The server identifies the product name and model number of the best matching antenna and generates the results in JSON format.

[0908] 8. The server sends the generated results to the user's device.

[0909] 9. The device analyzes the results received and displays to the user, "This is an antenna with model number XX."

[0910] Example prompts for generative AI models

[0911] Possible prompts include:

[0912] "You are developing a system to identify images of base station antennas. Follow these steps to describe a program that will analyze the image, match it against a database, and return the results."

[0913] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0914] Program processing steps

[0915] Step 1: User takes and submits image

[0916] The user takes an image of the components (e.g., antenna, radio, concrete pole) at the base station site using the smartphone camera. When taking the photo, the camera is adjusted so that the components are clearly visible. After taking the photo, the user launches a communication application on the smartphone (e.g., LINE, WhatsApp), selects the image, and sends it to the system.

[0917] Input: Image taken by the camera

[0918] Output: Image data sent to the system via a communication application

[0919] Step 2: Receiving and temporarily saving images on the device

[0920] The device receives images sent by the user via a communication application. The received image data is temporarily stored in internal memory. When saving, the device checks whether the image is in an appropriate format (e.g., JPEG, PNG), and if it is not, converts it to an appropriate format. It also checks that the image resolution is within a range suitable for analysis.

[0921] Input: Image data sent from a communication application

[0922] Output: Image data in the appropriate format, stored in the internal memory

[0923] Step 3: Sending images from the device to the server and analyzing them

[0924] The device generates a request to send pre-prepared image data to the server. This request also includes metadata (e.g., date and time of transmission, user ID). Once the request is sent, the server receives the request and saves the image data in a folder for internal processing. The server then passes the saved image data to the AI ​​module, which starts the image analysis process. The AI ​​module first pre-processes the image, removing noise and filtering, and then extracts feature information such as shape, text, and color.

[0925] Input: Image data stored in the internal memory

[0926] Output: Feature information extracted by the AI ​​module

[0927] Step 4: Server checks the feature information

[0928] The server compares the feature information extracted by the AI ​​module with the component information in the database. It uses a high-speed search algorithm (e.g., KD-Tree, Locality-Sensitive Hashing) to find the best match. It retrieves the product name, model number, and related details of the component from the database. If there are multiple candidates, it selects the best one.

[0929] Input: Feature information provided by the AI ​​module

[0930] Output: Component information extracted from the database (product name, model number, detailed information)

[0931] Step 5: Server generates and sends match results

[0932] The server generates a matching result to send to the user's device. This matching result is encoded as JSON data and includes necessary information (e.g., component product name, model number, and links to related documents). The server generates an HTTP response, embeds the JSON data in it, and sends it to the user's device.

[0933] Input: Material information extracted from the database

[0934] Output: Match results encoded in JSON format

[0935] Step 6: Display the match results on your device

[0936] The device analyzes the HTTP response received from the server and obtains the decoded JSON data. The device displays this data in a format that is easy for the user to understand. Specifically, the device displays a message such as "This is an antenna with model number XX" on the user screen, along with a link to more detailed information. The user can confirm this and check the detailed information as needed.

[0937] Input: JSON data of the match result sent from the server

[0938] Output: Material information displayed to the user

[0939] (Application example 1)

[0940] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0941] In factories, the task of identifying parts is typically manual and often time-consuming and labor-intensive. Furthermore, it is difficult for inexperienced workers to accurately grasp the information on identified parts, which can lead to problems due to misidentification. This can reduce work efficiency and negatively impact the productivity of the entire process. To solve these problems, a system that can quickly and effectively identify and manage parts is needed.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0943] In this invention, the server includes means for receiving images taken by a user, means for analyzing the images and extracting feature information, means for comparing the extracted feature information with information in a database, means for transmitting the comparison result to the user's terminal, means for the robot to take an image of a component, means for saving the image data in a built-in computer and transmitting the image data to an analysis module after converting its format, means for extracting feature information such as shape, text, and color using the image analysis module, and means for reporting component information to a management system using the extracted feature information. This enables the robot to automatically identify components in a factory and report information to the management system in real time.

[0944] "User" refers to a person who uses this system to take images and transmit data.

[0945] "Image" refers to still image data captured by a camera.

[0946] "Means for receiving" refers to devices or software that have the function of receiving and storing captured image data.

[0947] "Means for analyzing" refers to algorithms or software for extracting feature information from received image data.

[0948] "Feature information" refers to data such as shape, text, and color obtained through image analysis.

[0949] "Database" refers to a collection of information that stores detailed information about components.

[0950] "Means for matching" refers to a device or software that has the function of comparing extracted feature information with information in a database to find a match.

[0951] The "matching result" refers to the specific information of the component obtained by the matching means.

[0952] "Terminal" refers to an electronic device used by a user, such as a smartphone or computer.

[0953] "Robot" refers to a self-propelled mechanical device used to identify and manage parts within a factory.

[0954] "On-board computer" refers to a data processing device installed in a robot.

[0955] "Image analysis module" refers to a program or software for analyzing image data and extracting feature information.

[0956] "Management system" refers to a system for managing and displaying information about components within a factory.

[0957] This invention relates to a system in which a robot automatically identifies parts in a factory and reports the information to a management system in real time. This system implements a series of processes: receiving images taken by a user, analyzing them, extracting characteristic information, comparing them with a database, and sending the results. Furthermore, the system also includes functions for the robot to take images of parts, save the data in its built-in computer, send it to an analysis module, extract characteristic information, and report the part information to the management system.

[0958] The server has a means for receiving images taken by the user. For example, when a user takes an image with a smartphone and sends it via a communication application, the server receives the image and temporarily stores it.

[0959] Next, the server has a means to analyze the image and extract feature information. Here, an AI-based image analysis module is used, for example, the OpenCV library. This module extracts feature information such as shape, text, and color from the image data.

[0960] The server has a means for comparing the extracted feature information with information in a database. The database stores component information, and the server compares it with the information that best matches it.

[0961] The server has a means to send the matching results to the user's device. This means sends JSON format data to the user's device using an HTTP request. The user can check the results on their smartphone or other device.

[0962] The robot also has a means of taking images of the parts as it moves. The images are stored in the robot's onboard computer and converted to the appropriate format as needed. This computer then sends the image data to the analysis module, which then begins the image analysis.

[0963] The onboard computer uses an image analysis module to extract feature information such as shape, text, and color. Again, the OpenCV library can be used. The extracted feature information is reported to the management system.

[0964] For example, to identify new machine parts in a factory, a robot patrols a parts warehouse and takes pictures. The images are sent to a system, where they are analyzed and the results are reported to a management system. Workers can use this information to work more efficiently.

[0965] An example of a prompt for the generative AI model is, "Extract the feature information of the component from the image and compare it with the database. Identify the product name and model number of the corresponding component and display them to the user." Using this prompt, the AI ​​module can efficiently analyze the feature information and identify the component.

[0966] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0967] Step 1:

[0968] The user takes an image of a component using a smartphone or robot camera. The input is real-time image data obtained by the camera, and the output is an image file (e.g., JPEG, PNG).

[0969] Step 2:

[0970] The image taken by the user or robot is sent to the server via a communication application. Specifically, the user sends the image from their smartphone using a communication application (e.g., LINE, WhatsApp), and the robot similarly sends it to the server using its built-in communication module. The purpose of this step is for the image data to reach the server. The input is the captured image file, and the output is the sent image data.

[0971] Step 3:

[0972] The server temporarily stores the received image data. This storage process includes converting the image data to the appropriate file format (e.g., converting JPEG to PNG). The input is the raw image data received by the server, and the output is image data that has been appropriately saved in the temporary storage location, such as memory or storage.

[0973] Step 4:

[0974] The server passes the saved image data to the analysis module and begins image analysis. The analysis uses libraries such as OpenCV to extract feature information such as shape, text, and color from the image. The input is the temporarily saved image data, and the output is the extracted feature information. Specific operations include image filtering, text recognition, and shape analysis using OpenCV.

[0975] Step 5:

[0976] The server compares the extracted feature information with information in the database. Here, a comparison is made with pre-registered component information to identify the component information that best matches. The input is the extracted feature information and component information in the database, and the output is the specific information of the component that best matches (e.g., product name, model number).

[0977] Step 6:

[0978] The server generates matching results and sends them to the user's device. The generated matching results are encoded in a format such as JSON and sent to the user's device as an HTTP request. The input is information about the identified components, and the output is the encoded matching result data.

[0979] Step 7:

[0980] The terminal decodes the matching results received from the server and displays them to the user. The user can check the results on their smartphone or the management system. The input is the matching result data sent from the server, and the output is information displayed in a format that is easy for the user to understand. Specific operations include parsing JSON data and displaying the information on the UI.

[0981] Step 8:

[0982] The robot then patrols the parts in the factory, taking new images of the parts and sending them to the system. In this step, the robot's built-in computer stores the image data, passes it to an analysis module to extract feature information, and reports it to the management system. The input is the new image data taken by the robot, and the output is the part information reported to the management system.

[0983] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0984] This invention is a system for quickly and accurately identifying components in base station construction work, and aims to improve the user experience by recognizing the user's emotions and providing feedback based on those emotions. This allows even new employees and transferred employees to contribute to the work as an immediate asset.

[0985] System Overview

[0986] The system receives images taken by the user, analyzes them to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback on the results, achieving a series of steps. This allows the user to easily identify components, and the system recognizes and provides feedback on the user's emotions during the process, reducing stress for the user.

[0987] Program processing flow

[0988] 1. Taking and sending images

[0989] Users take pictures of components (e.g., antennas, radios, and concrete poles) at base station sites using their smartphone cameras, and then use communication applications (e.g., LINE or WhatsApp) to send the images to the system.

[0990] 2. Receiving and temporarily storing images

[0991] The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this point, the device converts the image into the appropriate format (e.g., JPEG, PNG) and prepares it for transmission to the server.

[0992] 3. Image submission and analysis

[0993] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[0994] 4. Matching feature information

[0995] The server compares the feature information extracted by the AI ​​module with the component information in the database, retrieves detailed information such as the product name and model number of the relevant component from the database, and selects the result with the best match.

[0996] 5. Emotion Recognition and Feedback

[0997] In parallel with image matching, the server runs an emotion engine that extracts emotional information from the user's facial image, voice, etc. The emotion engine recognizes the user's emotions (e.g., stress, anxiety, joy, etc.) and generates analysis results.

[0998] 6. Generating and transmitting emotion-based matching results

[0999] The server generates a match result adjusted based on the analysis results of the emotion engine and sends it to the user's device, which also includes a feedback message according to the user's emotion.

[1000] 7. Displaying matching results and feedback

[1001] The device decodes the response received from the server and displays the matching result and feedback in a format that is easy for the user to understand. The user can see the recognition result (e.g., "This is an antenna with model number XX") on the device along with a feedback message corresponding to the user's emotion.

[1002] Specific examples

[1003] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[1004] 2. The user sends the image to the system via a communication application.

[1005] 3. The device receives the image, converts it into the appropriate format, and sends it to the server.

[1006] 4. The server receives the image and passes it to the AI ​​module for analysis.

[1007] 5. The AI ​​module extracts the antenna's characteristic information and compares it with a database.

[1008] 6. The server identifies the product name and model number of the matching antenna, and in parallel the emotion engine recognizes the user's emotion.

[1009] 7. The server generates feedback based on the matching results and the user's emotions and sends it to the user's device as JSON format data.

[1010] 8. The device displays the matching results and feedback to the user, notifying them with a message such as "This is an antenna with model number XX" and "Good job, almost done."

[1011] Through this process, the system can help users quickly identify parts on-site, while also recognizing users' emotions and providing appropriate feedback to improve the user experience.

[1012] The processing flow will be explained below.

[1013] Step 1:

[1014] The user takes pictures of components (e.g., antennas, radios, and concrete poles) at the base station site using the smartphone camera.

[1015] Step 2:

[1016] The user opens a communication application (e.g., LINE, WhatsApp), selects the captured image, and presses the "Send" button to send it to the system.

[1017] Step 3:

[1018] The device receives the image via the communication application, temporarily stores the image data in its internal memory, and converts the image to an appropriate format (e.g., JPEG, PNG).

[1019] Step 4:

[1020] The device generates an HTTP POST request to send the saved image data to the server, and sends it to the server's image reception endpoint.

[1021] Step 5:

[1022] The server receives the HTTP request, extracts the image data from the request body, and temporarily stores it.

[1023] Step 6:

[1024] The server passes the stored image to the AI ​​module and begins image analysis.

[1025] Step 7:

[1026] The AI ​​module (in the server) analyzes the image and extracts feature information such as shape, text, and color.

[1027] Step 8:

[1028] Based on the feature information extracted by the AI ​​module, the server generates an SQL query to match it with component information in the database.

[1029] Step 9:

[1030] The server runs an SQL query against the database to obtain detailed information such as the product name and model number of the relevant part.

[1031] Step 10:

[1032] The server generates the matching results and simultaneously runs an emotion engine that extracts emotion information from the user's facial image, voice, etc.

[1033] Step 11:

[1034] The emotion engine (in the server) recognizes the user's emotions (e.g., stress, anxiety, joy, etc.) and generates the analysis results.

[1035] Step 12:

[1036] The server generates an adjusted matching result based on the analysis results of the emotion engine and sends it to the user's device, which also includes a feedback message according to the user's emotion.

[1037] Step 13:

[1038] The device receives the response from the server and decodes the JSON data.

[1039] Step 14:

[1040] The terminal displays the recognition result (e.g., "This is an antenna with model number XX") to the user, as well as a feedback message according to the user's emotion (e.g., "Good job, you're almost done").

[1041] Step 15:

[1042] The user checks the recognition results and feedback messages displayed on the terminal, and if necessary, proceeds with further investigation or response.

[1043] This specific step allows the system to help users quickly identify components in the field, and also recognizes the user's emotions and provides appropriate feedback.

[1044] Example 2

[1045] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1046] In base station construction work, there is a need for a method that allows even new employees or newly transferred employees to quickly and accurately identify components. There is also a need for a feedback function to reduce the stress and anxiety of users who work long hours. There has not been a system that meets these needs, so there is a need to solve these issues.

[1047] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1048] In this invention, the server includes means for receiving an image taken by a user, means for analyzing the image and extracting feature information, means for comparing the extracted feature information with information in a database, means for recognizing the user's emotion, means for generating feedback based on the emotion recognition result, and means for transmitting the generated feedback to the user's terminal. This allows the user to easily identify components and receive appropriate feedback regarding changes in emotion during the process, thereby improving work efficiency and the user experience.

[1049] The "means for receiving images taken by a user" refers to a device or method for acquiring image data taken by a user with a photographing device such as a smartphone through communication.

[1050] "Means for analyzing images and extracting feature information" refers to a device or method that analyzes captured image data using machine learning algorithms or image processing techniques to extract features such as shape, text, and color within the image.

[1051] The "means for matching with information in a database" refers to a device or method that searches for items that match information in a pre-registered database based on the extracted characteristic information and generates a matching result.

[1052] "Means for recognizing user emotions" refers to a device or method for analyzing a user's facial expressions and tone of voice from image or audio data, etc., to detect the user's emotional state (e.g., stress, anxiety, joy, etc.).

[1053] The "means for generating feedback based on emotion recognition results" refers to a device or method for creating an appropriate feedback message according to the recognized emotional state of the user and notifying the user of the appropriate feedback message.

[1054] The "means for transmitting the generated feedback to the user's terminal" refers to a device or method for transmitting the generated feedback message from the server to the user's terminal such as a smartphone or tablet via data communication.

[1055] This invention is a system for quickly and accurately identifying components in base station construction work. It aims to improve the user experience by recognizing the user's emotions and providing feedback based on those emotions. This allows even new employees and newly transferred employees to contribute immediately to the work. The system receives images taken by the user, analyzes the images to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback based on the results.

[1056] The specific configuration of the system is as follows.

[1057] 1. Receiving images

[1058] The user uses the smartphone camera to take an image of the component at the base station site. The captured image is then sent to the system using a communication application (e.g., LINE or WhatsApp). The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this time, the device converts the image into an appropriate format (e.g., JPEG or PNG).

[1059] 2. Sending images and preparing for analysis

[1060] The device generates a request to send the stored image data to the server and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. This AI module performs image analysis using a deep learning framework (e.g., TensorFlow, PyTorch).

[1061] 3. Extraction of feature information

[1062] The server uses an AI module to extract feature information such as shape, text, and color from the image, which is then used for database matching.

[1063] 4. Database Matching

[1064] The server compares the extracted feature information with the registered information in the database. The database is built using a common RDBMS (e.g., MySQL or PostgreSQL) and stores detailed information such as the product name and model number of the component. The server selects the most matching result and temporarily stores it.

[1065] 5. Performing Emotion Recognition

[1066] In parallel, the emotion engine extracts emotional information from the user's facial image, voice, and other data. The emotion engine performs emotion analysis using, for example, OpenCV or a natural language processing (NLP) model. The recognized emotional data (e.g., stress level, anxiety level, happiness level) is saved.

[1067] 6. Producing adjusted matching results

[1068] The server adds a feedback message to the matching result based on the emotion recognition results. For example, if the user is feeling stressed, it generates a message such as "Good job, you're almost done." This data is converted into JSON format and sent to the user's device.

[1069] 7. Viewing Results and Feedback

[1070] The device receives the response from the server and displays the analysis results and feedback on the user interface. The user can check the recognition result (e.g., "This is an antenna with model number XX") and the emotional feedback message on the device.

[1071] Specific examples

[1072] A user takes a picture of the newly installed antenna of the base station with their smartphone.

[1073] The user uses a communication application to send the image to the system.

[1074] The device receives the image, converts it into the appropriate format, and sends it to the server.

[1075] The server receives the images and analyzes them using an AI module.

[1076] The AI ​​module extracts characteristic information and compares it with a database.

[1077] The server identifies the matching component information, and in parallel, the emotion engine recognizes the user's emotion.

[1078] The server generates adjusted matching results and emotional feedback and sends them to the device.

[1079] The device will display the results, along with a message such as "This is an antenna with model number XX" and "Thank you for your hard work, it's almost done."

[1080] Prompt Sentence Examples

[1081] "I would like to develop a system that analyzes and identifies images of components taken at the base station's work site and returns the results. It also needs a function that recognizes the user's emotions and provides feedback. Please explain in detail the program flow for such a system."

[1082] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1083] Step 1:

[1084] The user uses the smartphone camera to take pictures of components (e.g., antennas, radios, and concrete poles) at the base station site. The input is the image taken by the user with the camera, and the output is an image file saved on the smartphone. Specifically, the user launches the smartphone's camera app, frames the component, and presses the capture button.

[1085] Step 2:

[1086] The user sends the captured image to the system using a communication application (e.g., LINE or WhatsApp). The input is an image file stored on the smartphone, and the output is image data sent through the communication application. Specifically, the user opens the communication application, displays the chat screen, attaches the image, and presses the send button.

[1087] Step 3:

[1088] The device receives images via a communication application and temporarily stores the image data in its internal memory. The input is the image data received from the communication application, and the output is an image file stored in a temporary storage folder. Specifically, the device receives an application notification, downloads the image data, and stores it in a temporary storage folder (e.g., / tmp / images).

[1089] Step 4:

[1090] The device converts the received image into an appropriate format (e.g., JPEG, PNG) and generates a request to send to the server. The input is the image file saved in the temporary storage folder, and the output is the image data sent to the server as an HTTP POST request. Specifically, the device checks the image format, performs conversion processing if necessary, and generates an HTTP POST request.

[1091] Step 5:

[1092] The server passes the received image data to the AI ​​module and begins image analysis. The input is the image data received as an HTTP POST request, and the output is the image data waiting to be analyzed. Specifically, the server obtains the image data via the file reception API and passes it to the AI ​​module.

[1093] Step 6:

[1094] The server uses an AI module to extract feature information such as shape, text, and color from images. The input is the image data passed to the AI ​​module, and the output is the extracted feature information. Specifically, the server runs an AI module (e.g., TensorFlow, PyTorch) and applies an image analysis algorithm to extract feature information.

[1095] Step 7:

[1096] The server compares the extracted feature information with the component information in the database. The input is the extracted feature information, and the output is the component information as a comparison result. Specifically, the server generates a database query and searches a database such as MySQL or PostgreSQL to obtain the most matching component information.

[1097] Step 8:

[1098] The server also uses an emotion engine to recognize the user's emotions. The input is the user's facial image and voice data, and the output is the recognized emotion data. Specifically, the server runs the emotion engine (e.g., OpenCV, NLP model) to extract emotion information from the facial image and voice.

[1099] Step 9:

[1100] The server generates an adjusted matching result and a feedback message based on the emotion recognition result. The input is the component information as the matching result and the recognized emotion data, and the output is the adjusted matching result and a feedback message. Specifically, the server creates a feedback message based on the emotion data and converts it into JSON format.

[1101] Step 10:

[1102] The server sends the generated feedback message to the user's device. The input is the matching result and feedback message converted to JSON format, and the output is the data sent to the user's device as an HTTP response. Specifically, the server sends the JSON data through an API endpoint.

[1103] Step 11:

[1104] The terminal receives the response from the server and displays the analysis results and feedback on the user interface. The input is the JSON data received from the server, and the output is the recognition results and feedback messages displayed on the user interface. Specifically, the terminal parses the JSON data and displays it in a format that is easy for the user to understand.

[1105] Specific examples

[1106] A user takes a picture of the newly installed antenna of the base station with their smartphone.

[1107] The user uses a communication application to send the image to the system.

[1108] The device receives the image, converts it into the appropriate format, and sends it to the server.

[1109] The server receives the images and analyzes them using an AI module.

[1110] The AI ​​module extracts characteristic information and compares it with a database.

[1111] The server identifies the matching component information, and in parallel, the emotion engine recognizes the user's emotion.

[1112] The server generates adjusted matching results and emotional feedback and sends them to the device.

[1113] The device will display the results, along with a message such as "This is an antenna with model number XX" and "Thank you for your hard work, it's almost done."

[1114] (Application example 2)

[1115] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1116] Conventional parts identification work in factories often relies on the experience and knowledge of the worker, making it difficult for new employees or newly transferred workers to quickly and accurately identify parts. Furthermore, there was no mechanism to reduce stress and anxiety during work, leading to calls for improvements to the work environment. To solve these issues, a system is needed that automates the parts identification process and improves work efficiency by providing feedback on the user's emotions.

[1117] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving an image taken by a user, means for analyzing the image and extracting feature information, means for comparing the extracted feature information with information in a database, means for analyzing the user's emotions using an emotion engine, and means for generating a feedback message based on the analyzed emotion information. This allows factory workers to easily identify components, and by receiving feedback that takes into account their emotions during work, it is possible to reduce stress and improve work efficiency.

[1118] "User" refers to the factory worker who takes and sends images to the system for part identification and emotion recognition.

[1119] "Means for receiving images" refers to a communication function for allowing the server to receive images taken by the user.

[1120] The "means for analyzing an image and extracting characteristic information" refers to an image analysis function for extracting characteristic information such as the shape, text, and color of a component from a received image.

[1121] "Database" refers to data storage for storing component information and feature data.

[1122] "Means for matching" refers to a computer program for comparing extracted characteristic information with information in a database and identifying matching components.

[1123] "User's device" refers to the device used to receive and display matching results and feedback messages, specifically a smartphone, tablet, or PC.

[1124] An "emotion engine" refers to an algorithm or software that analyzes facial images and voice data taken by a user to generate emotional information.

[1125] "Emotion information" is data that indicates the user's emotional state, and includes emotions such as stress, anxiety, and joy.

[1126] A "feedback message" refers to a response message provided to a user based on emotional information.

[1127] "System" refers to the collection of devices and software that integrates and operates all of these means.

[1128] The present invention is a system for improving the efficiency of component identification work in a factory and for improving the work environment by providing feedback on the emotions of workers. Specific embodiments for carrying out the invention are described below.

[1129] System Overview

[1130] The system receives images taken by the user, analyzes them to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback based on the results. This allows workers to easily identify parts and receive feedback based on their emotions.

[1131] Hardware and software used

[1132] Hardware

[1133] Smartphone terminal: Used by factory workers to take and send images of parts.

[1134] Robot: Used to move autonomously within the factory and take images of components.

[1135] Server: Performs image analysis, database matching, emotion recognition, and feedback message generation.

[1136] software

[1137] Image analysis: Extracting feature information from images using TensorFlow and OpenCV.

[1138] Database: MySQL is used to store and manage component information.

[1139] Emotion Recognition: Analyze user emotions using Azure Cognitive Services and Google Cloud Natural Language API.

[1140] Communication applications: Use common communication applications (e.g., LINE or WhatsApp) to send images.

[1141] System operation explanation

[1142] Taking and sending images

[1143] The user takes a picture of the component using a smartphone or a camera mounted on the robot and sends it to the server via a communication application.

[1144] Image analysis and feature information extraction

[1145] The server analyzes the received images using TensorFlow and OpenCV to extract feature information such as shape, text, and color, allowing it to compare detailed information about the identified components with a database.

[1146] Emotion Recognition and Feedback

[1147] In parallel, the server analyzes the user's facial image and voice data using Azure Cognitive Services and Google Cloud Natural Language API to generate emotional information, and then generates a feedback message based on the results and sends it to the user's device.

[1148] Specific examples

[1149] For example, suppose a user takes a picture of a component (e.g., a screw) in a factory using their smartphone. The image is sent to a server via a communication application (e.g., LINE or WhatsApp). The server analyzes the image, extracts the component's characteristics, and compares them with a database. The emotion engine then analyzes the user's emotions and generates a feedback message such as "This screw is model number XX" or "Thank you for your hard work. The work is going well," which is then sent to the user's device.

[1150] Examples of prompt statements

[1151] image_path = 'path_to_image'

[1152] user_data = 'Information including voice data and user facial images'

[1153] result = main(image_path, user_data)

[1154] print(json.dumps(result, indent=4, ensure_ascii=False))

[1155] In this way, the present invention allows factory workers to quickly and accurately identify parts and contributes to improving the work environment by providing feedback on their emotions while working.

[1156] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1157] Step 1:

[1158] The user takes an image of the part. Using a smartphone or a camera mounted on a robot, the image of the part is captured. This generates image data, which then becomes the input for processing in the next step.

[1159] Step 2:

[1160] The user sends the image they have taken to their device via a communication application, such as LINE or WhatsApp. During this process, the image data is sent from the smartphone to the server.

[1161] Step 3:

[1162] The server temporarily stores the images received from the user. The image data is converted into an appropriate format (JPEG, PNG, etc.) and stored in the internal memory. This stored image data is used in the next analysis step.

[1163] Step 4:

[1164] The server analyzes the stored image data using TensorFlow and OpenCV. Image analysis extracts feature information such as shape, text, and color. This feature information becomes input data for matching with the database.

[1165] Step 5:

[1166] The server compares the extracted feature information with the component information in the database, which stores information such as component model numbers and names. As a result of the comparison, the component information that best matches the information is identified.

[1167] Step 6:

[1168] The server simultaneously collects the user's emotional information and analyzes it using Azure Cognitive Services and Google Cloud Natural Language API. Facial images and voice data are input as the user's emotional data, and this emotional data is analyzed to output emotional information such as stress, anxiety, and joy.

[1169] Step 7:

[1170] The server generates a feedback message based on the analysis results of the emotion engine. An appropriate feedback message is generated based on the component information of the collation result and the user's emotion information. This feedback message is then sent to the user terminal.

[1171] Step 8:

[1172] The terminal displays the matching results and feedback messages received from the server to the user. The user can check information on the terminal, such as "This screw is model number XX" or "Thank you for your hard work, the work is going well." This improves work efficiency and the user experience.

[1173] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1174] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1175] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1176] [Fourth embodiment]

[1177] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1178] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1179] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1180] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1181] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1182] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1183] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1184] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1185] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1186] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1187] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1188] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1189] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1190] This invention is a system for quickly and accurately identifying materials in base station construction work, allowing even new employees and transferred employees to contribute immediately to the work.

[1191] System Overview

[1192] The system receives images taken by the user, analyzes them to extract characteristic information, compares them with a database, and returns the results to the user, allowing the user to easily identify components.

[1193] Program processing flow

[1194] 1. Taking and sending images

[1195] Users take pictures of components (e.g., antennas, radios, and concrete poles) at base station sites using their smartphone cameras, and then use communication applications (e.g., LINE, WhatsApp) to send the images to the system.

[1196] 2. Receiving and temporary storage of images

[1197] The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this point, the device converts the image into the appropriate format (e.g., JPEG, PNG) and prepares it for transmission to the server.

[1198] 3. Image submission and analysis

[1199] The device generates a request to send the stored image data to the server, and sends it to the server. The server passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[1200] 4. Matching feature information

[1201] The server compares the feature information extracted by the AI ​​module with the component information in the database, retrieves detailed information such as the product name and model number of the relevant component from the database, and selects the result with the best match.

[1202] 5. Generating and sending matching results

[1203] The server generates a match result to send to the user's device, encodes it as JSON data, and sends this data to the user's device as an HTTP response.

[1204] 6. Display of matching results

[1205] The device decodes the response received from the server and displays the matching result in a format that is easy for the user to understand. The user can then confirm the recognition result (e.g., "This is an antenna with model number XX") on the device.

[1206] Specific examples

[1207] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[1208] 2. The user sends the image to the system via a communication application.

[1209] 3. The device receives the image, converts it into the appropriate format, and sends it to the server.

[1210] 4. The server receives the image and passes it to the AI ​​module for analysis.

[1211] 5. The AI ​​module extracts the antenna's characteristic information and compares it with a database.

[1212] 6. The server identifies the product name and model number of the matching antenna and generates a match.

[1213] 7. The server sends the matching results to the user's device.

[1214] 8. The device displays the matching results to the user and notifies them that the antenna is model number XX.

[1215] Through this process, the system enables users to quickly identify components on-site, improving the efficiency and accuracy of on-site operations.

[1216] The processing flow will be explained below.

[1217] Step 1:

[1218] The user takes pictures of components (e.g., antennas, radios, and concrete poles) at the base station site using their smartphone camera.

[1219] Step 2:

[1220] The user opens a communication application (e.g., LINE, WhatsApp), selects the captured image, and presses the "Send" button to send it to the system.

[1221] Step 3:

[1222] The device receives the image via the communication application, temporarily stores the image data in its internal memory, and converts the image to an appropriate format (e.g., JPEG, PNG).

[1223] Step 4:

[1224] The device generates an HTTP POST request to send the saved image data to the server, and sends it to the server's image reception endpoint.

[1225] Step 5:

[1226] The server receives the HTTP request, extracts the image data from the request body, and temporarily stores it.

[1227] Step 6:

[1228] The server passes the stored image to the AI ​​module and begins image analysis.

[1229] Step 7:

[1230] The AI ​​module (in the server) analyzes the image and extracts feature information such as shape, text, and color.

[1231] Step 8:

[1232] Based on the feature information extracted by the AI ​​module, the server generates an SQL query to match it with component information in the database.

[1233] Step 9:

[1234] The server runs an SQL query against the database to obtain detailed information such as the product name and model number of the relevant part.

[1235] Step 10:

[1236] The server selects the best match, encodes the recognition result in JSON format, stores it in an HTTP response, and sends it to the user's device.

[1237] Step 11:

[1238] The device receives the response from the server and decodes the JSON data.

[1239] Step 12:

[1240] The device updates the GUI to display the recognition result (e.g., "This is an antenna with model number XX") to the user.

[1241] Step 13:

[1242] The user checks the recognition results displayed on the terminal and, if necessary, proceeds with further investigation or response.

[1243] Example 1

[1244] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1245] Identifying materials in base station construction work requires engineers with specialized knowledge, and it is difficult for new employees or transferred employees to quickly and accurately identify materials on-site. This can reduce work efficiency and affect accuracy. Furthermore, there is a risk of frequent mistakes and rework on-site, so a solution to these issues is needed.

[1246] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1247] In this invention, the server includes a device that receives images taken by a user, an analysis device that analyzes the images and extracts feature information, a comparison device that compares the extracted feature information with information in a database, and a communication device that transmits the comparison results to the user's device, thereby enabling the user to quickly and accurately identify components based on images taken at the site.

[1248] A "user" is a person who uses the system to identify components.

[1249] An "image" is a file that represents visual information of an object photographed by a user.

[1250] A "device" is a device for receiving images taken by a user.

[1251] An "analysis device" is a device that analyzes a received image and extracts feature information.

[1252] "Feature information" is discernible data such as shape, text, and color extracted from an image.

[1253] A "database" is a digital storage device that stores information to be matched with characteristic information.

[1254] A "matching device" is a device that matches the extracted feature information with information in a database.

[1255] A "communication device" is a device for transmitting the collation results to the user's device.

[1256] "Communication software" is a program for transmitting images taken by the user.

[1257] The "display device" is a device for displaying detailed information about a component based on its characteristic information.

[1258] The "system" is a set of devices and software that identifies components based on images taken by the user.

[1259] This invention relates to a system for quickly and accurately identifying components in base station construction work. The system analyzes images taken by users on-site, compares them with a database, and provides information on the relevant components. The specific system configuration, the hardware and software used, and the data processing method are described in detail below.

[1260] System configuration

[1261] The system mainly consists of the following components:

[1262] 1. Your device

[1263] A mobile device such as a smartphone or tablet is used, which has a camera and communication software (e.g., LINE, WhatsApp) installed.

[1264] 2. Analysis device

[1265] It uses an AI module located on the server, specifically consisting of hardware and software (e.g., TensorFlow, PyTorch) for executing image recognition algorithms.

[1266] 3. Database

[1267] A central database that stores component information, including detailed information such as component product name, model number, shape, and color.

[1268] 4. Communications Equipment

[1269] It includes the network infrastructure (e.g. HTTP server, RESTful API) for communication between the server and the user's device.

[1270] Specific program processing and usage

[1271] 1. Taking and sending images

[1272] The user takes an image of the component at the base station site using the smartphone camera, then launches the communication software on the smartphone and sends the image to the system.

[1273] 2. Receiving and temporarily storing images

[1274] The device receives the image via the communication software, temporarily stores it in its internal memory, and converts it into the appropriate format (e.g., JPEG, PNG) as needed.

[1275] 3. Image submission and analysis

[1276] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[1277] 4. Matching feature information

[1278] The server compares the feature information extracted by the AI ​​module with the component information in the database, selects the most matching result, and obtains detailed information.

[1279] 5. Generating and sending matching results

[1280] The server generates a match result to send to the user's device, encodes it as JSON data, and sends it to the user's device as an HTTP response.

[1281] 6. Display of matching results

[1282] The device decodes the response received from the server and displays the matching results in a format that is easy for the user to understand, such as a message like "This is an antenna with model number XX."

[1283] Specific examples

[1284] For example:

[1285] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[1286] 2. The user selects an image in the LINE app and sends it to the system.

[1287] 3. The device receives the image, converts it to JPEG format, and saves it to its internal memory.

[1288] 4. The device sends the saved image to the server.

[1289] 5. The server receives the image and passes it to the AI ​​module for analysis.

[1290] 6. The AI ​​module extracts features such as antenna shape, text, and color and compares them with a database.

[1291] 7. The server identifies the product name and model number of the best matching antenna and generates the results in JSON format.

[1292] 8. The server sends the generated results to the user's device.

[1293] 9. The device analyzes the results received and displays to the user, "This is an antenna with model number XX."

[1294] Example prompts for generative AI models

[1295] Possible prompts include:

[1296] "You are developing a system to identify images of base station antennas. Follow these steps to describe a program that will analyze the image, match it against a database, and return the results."

[1297] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1298] Program processing steps

[1299] Step 1: User takes and submits image

[1300] The user takes an image of the components (e.g., antenna, radio, concrete pole) at the base station site using the smartphone camera. When taking the photo, the camera is adjusted so that the components are clearly visible. After taking the photo, the user launches a communication application on the smartphone (e.g., LINE, WhatsApp), selects the image, and sends it to the system.

[1301] Input: Image taken by the camera

[1302] Output: Image data sent to the system via a communication application

[1303] Step 2: Receiving and temporarily saving images on the device

[1304] The device receives images sent by the user via a communication application. The received image data is temporarily stored in internal memory. When saving, the device checks whether the image is in an appropriate format (e.g., JPEG, PNG), and if it is not, converts it to an appropriate format. It also checks that the image resolution is within a range suitable for analysis.

[1305] Input: Image data sent from a communication application

[1306] Output: Image data in the appropriate format, stored in the internal memory

[1307] Step 3: Sending images from the device to the server and analyzing them

[1308] The device generates a request to send pre-prepared image data to the server. This request also includes metadata (e.g., date and time of transmission, user ID). Once the request is sent, the server receives the request and saves the image data in a folder for internal processing. The server then passes the saved image data to the AI ​​module, which starts the image analysis process. The AI ​​module first pre-processes the image, removing noise and filtering, and then extracts feature information such as shape, text, and color.

[1309] Input: Image data stored in the internal memory

[1310] Output: Feature information extracted by the AI ​​module

[1311] Step 4: Server checks the feature information

[1312] The server compares the feature information extracted by the AI ​​module with the component information in the database. It uses a high-speed search algorithm (e.g., KD-Tree, Locality-Sensitive Hashing) to find the best match. It retrieves the product name, model number, and related details of the component from the database. If there are multiple candidates, it selects the best one.

[1313] Input: Feature information provided by the AI ​​module

[1314] Output: Component information extracted from the database (product name, model number, detailed information)

[1315] Step 5: Server generates and sends match results

[1316] The server generates a matching result to send to the user's device. This matching result is encoded as JSON data and includes necessary information (e.g., component product name, model number, and links to related documents). The server generates an HTTP response, embeds the JSON data in it, and sends it to the user's device.

[1317] Input: Material information extracted from the database

[1318] Output: Match results encoded in JSON format

[1319] Step 6: Display the match results on your device

[1320] The device analyzes the HTTP response received from the server and obtains the decoded JSON data. The device displays this data in a format that is easy for the user to understand. Specifically, the device displays a message such as "This is an antenna with model number XX" on the user screen, along with a link to more detailed information. The user can confirm this and check the detailed information as needed.

[1321] Input: JSON data of the match result sent from the server

[1322] Output: Material information displayed to the user

[1323] (Application example 1)

[1324] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1325] In factories, the task of identifying parts is typically manual and often time-consuming and labor-intensive. Furthermore, it is difficult for inexperienced workers to accurately grasp the information on identified parts, which can lead to problems due to misidentification. This can reduce work efficiency and negatively impact the productivity of the entire process. To solve these problems, a system that can quickly and effectively identify and manage parts is needed.

[1326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1327] In this invention, the server includes means for receiving images taken by a user, means for analyzing the images and extracting feature information, means for comparing the extracted feature information with information in a database, means for transmitting the comparison result to the user's terminal, means for the robot to take an image of a component, means for saving the image data in a built-in computer and transmitting the image data to an analysis module after converting its format, means for extracting feature information such as shape, text, and color using the image analysis module, and means for reporting component information to a management system using the extracted feature information. This enables the robot to automatically identify components in a factory and report information to the management system in real time.

[1328] "User" refers to a person who uses this system to take images and transmit data.

[1329] "Image" refers to still image data captured by a camera.

[1330] "Means for receiving" refers to devices or software that have the function of receiving and storing captured image data.

[1331] "Means for analyzing" refers to algorithms or software for extracting feature information from received image data.

[1332] "Feature information" refers to data such as shape, text, and color obtained through image analysis.

[1333] "Database" refers to a collection of information that stores detailed information about components.

[1334] "Means for matching" refers to a device or software that has the function of comparing extracted feature information with information in a database to find a match.

[1335] The "matching result" refers to the specific information of the component obtained by the matching means.

[1336] "Terminal" refers to an electronic device used by a user, such as a smartphone or computer.

[1337] "Robot" refers to a self-propelled mechanical device used to identify and manage parts within a factory.

[1338] "On-board computer" refers to a data processing device installed in a robot.

[1339] "Image analysis module" refers to a program or software for analyzing image data and extracting feature information.

[1340] "Management system" refers to a system for managing and displaying information about components within a factory.

[1341] This invention relates to a system in which a robot automatically identifies parts in a factory and reports the information to a management system in real time. This system implements a series of processes: receiving images taken by a user, analyzing them, extracting characteristic information, comparing them with a database, and sending the results. Furthermore, the system also includes functions for the robot to take images of parts, save the data in its built-in computer, send it to an analysis module, extract characteristic information, and report the part information to the management system.

[1342] The server has a means for receiving images taken by the user. For example, when a user takes an image with a smartphone and sends it via a communication application, the server receives the image and temporarily stores it.

[1343] Next, the server has a means to analyze the image and extract feature information. Here, an AI-based image analysis module is used, for example, the OpenCV library. This module extracts feature information such as shape, text, and color from the image data.

[1344] The server has a means for comparing the extracted feature information with information in a database. The database stores component information, and the server compares it with the information that best matches it.

[1345] The server has a means to send the matching results to the user's device. This means sends JSON format data to the user's device using an HTTP request. The user can check the results on their smartphone or other device.

[1346] The robot also has a means of taking images of the parts as it moves. The images are stored in the robot's onboard computer and converted to the appropriate format as needed. This computer then sends the image data to the analysis module, which then begins the image analysis.

[1347] The onboard computer uses an image analysis module to extract feature information such as shape, text, and color. Again, the OpenCV library can be used. The extracted feature information is reported to the management system.

[1348] For example, to identify new machine parts in a factory, a robot patrols a parts warehouse and takes pictures. The images are sent to a system, where they are analyzed and the results are reported to a management system. Workers can use this information to work more efficiently.

[1349] An example of a prompt for the generative AI model is, "Extract the feature information of the component from the image and compare it with the database. Identify the product name and model number of the corresponding component and display them to the user." Using this prompt, the AI ​​module can efficiently analyze the feature information and identify the component.

[1350] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1351] Step 1:

[1352] The user takes an image of a component using a smartphone or robot camera. The input is real-time image data obtained by the camera, and the output is an image file (e.g., JPEG, PNG).

[1353] Step 2:

[1354] The image taken by the user or robot is sent to the server via a communication application. Specifically, the user sends the image from their smartphone using a communication application (e.g., LINE, WhatsApp), and the robot similarly sends it to the server using its built-in communication module. The purpose of this step is for the image data to reach the server. The input is the captured image file, and the output is the sent image data.

[1355] Step 3:

[1356] The server temporarily stores the received image data. This storage process includes converting the image data to the appropriate file format (e.g., converting JPEG to PNG). The input is the raw image data received by the server, and the output is image data that has been appropriately saved in the temporary storage location, such as memory or storage.

[1357] Step 4:

[1358] The server passes the saved image data to the analysis module and begins image analysis. The analysis uses libraries such as OpenCV to extract feature information such as shape, text, and color from the image. The input is the temporarily saved image data, and the output is the extracted feature information. Specific operations include image filtering, text recognition, and shape analysis using OpenCV.

[1359] Step 5:

[1360] The server compares the extracted feature information with information in the database. Here, a comparison is made with pre-registered component information to identify the component information that best matches. The input is the extracted feature information and component information in the database, and the output is the specific information of the component that best matches (e.g., product name, model number).

[1361] Step 6:

[1362] The server generates matching results and sends them to the user's device. The generated matching results are encoded in a format such as JSON and sent to the user's device as an HTTP request. The input is information about the identified components, and the output is the encoded matching result data.

[1363] Step 7:

[1364] The terminal decodes the matching results received from the server and displays them to the user. The user can check the results on their smartphone or the management system. The input is the matching result data sent from the server, and the output is information displayed in a format that is easy for the user to understand. Specific operations include parsing JSON data and displaying the information on the UI.

[1365] Step 8:

[1366] The robot then patrols the parts in the factory, taking new images of the parts and sending them to the system. In this step, the robot's built-in computer stores the image data, passes it to an analysis module to extract feature information, and reports it to the management system. The input is the new image data taken by the robot, and the output is the part information reported to the management system.

[1367] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1368] This invention is a system for quickly and accurately identifying components in base station construction work, and aims to improve the user experience by recognizing the user's emotions and providing feedback based on those emotions. This allows even new employees and transferred employees to contribute to the work as an immediate asset.

[1369] System Overview

[1370] The system receives images taken by the user, analyzes them to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback on the results, achieving a series of steps. This allows the user to easily identify components, and the system recognizes and provides feedback on the user's emotions during the process, reducing stress for the user.

[1371] Program processing flow

[1372] 1. Taking and sending images

[1373] Users take pictures of components (e.g., antennas, radios, and concrete poles) at base station sites using their smartphone cameras, and then use communication applications (e.g., LINE or WhatsApp) to send the images to the system.

[1374] 2. Receiving and temporarily storing images

[1375] The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this point, the device converts the image into the appropriate format (e.g., JPEG, PNG) and prepares it for transmission to the server.

[1376] 3. Image submission and analysis

[1377] The device generates a request to send the stored image data to the server, and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. The AI ​​module analyzes the image and extracts feature information such as shape, text, and color.

[1378] 4. Matching feature information

[1379] The server compares the feature information extracted by the AI ​​module with the component information in the database, retrieves detailed information such as the product name and model number of the relevant component from the database, and selects the result with the best match.

[1380] 5. Emotion Recognition and Feedback

[1381] In parallel with image matching, the server runs an emotion engine that extracts emotional information from the user's facial image, voice, etc. The emotion engine recognizes the user's emotions (e.g., stress, anxiety, joy, etc.) and generates analysis results.

[1382] 6. Generating and transmitting emotion-based matching results

[1383] The server generates a match result adjusted based on the analysis results of the emotion engine and sends it to the user's device, which also includes a feedback message according to the user's emotion.

[1384] 7. Displaying matching results and feedback

[1385] The device decodes the response received from the server and displays the matching result and feedback in a format that is easy for the user to understand. The user can see the recognition result (e.g., "This is an antenna with model number XX") on the device along with a feedback message corresponding to the user's emotion.

[1386] Specific examples

[1387] 1. A user takes a picture of the base station's newly installed antenna with their smartphone.

[1388] 2. The user sends the image to the system via a communication application.

[1389] 3. The device receives the image, converts it into the appropriate format, and sends it to the server.

[1390] 4. The server receives the image and passes it to the AI ​​module for analysis.

[1391] 5. The AI ​​module extracts the antenna's characteristic information and compares it with a database.

[1392] 6. The server identifies the product name and model number of the matching antenna, and in parallel the emotion engine recognizes the user's emotion.

[1393] 7. The server generates feedback based on the matching results and the user's emotions and sends it to the user's device as JSON format data.

[1394] 8. The device displays the matching results and feedback to the user, notifying them with a message such as "This is an antenna with model number XX" and "Good job, almost done."

[1395] Through this process, the system can help users quickly identify parts on-site, while also recognizing users' emotions and providing appropriate feedback to improve the user experience.

[1396] The processing flow will be explained below.

[1397] Step 1:

[1398] The user takes pictures of components (e.g., antennas, radios, and concrete poles) at the base station site using the smartphone camera.

[1399] Step 2:

[1400] The user opens a communication application (e.g., LINE, WhatsApp), selects the captured image, and presses the "Send" button to send it to the system.

[1401] Step 3:

[1402] The device receives the image via the communication application, temporarily stores the image data in its internal memory, and converts the image to an appropriate format (e.g., JPEG, PNG).

[1403] Step 4:

[1404] The device generates an HTTP POST request to send the saved image data to the server, and sends it to the server's image reception endpoint.

[1405] Step 5:

[1406] The server receives the HTTP request, extracts the image data from the request body, and temporarily stores it.

[1407] Step 6:

[1408] The server passes the stored image to the AI ​​module and begins image analysis.

[1409] Step 7:

[1410] The AI ​​module (in the server) analyzes the image and extracts feature information such as shape, text, and color.

[1411] Step 8:

[1412] Based on the feature information extracted by the AI ​​module, the server generates an SQL query to match it with component information in the database.

[1413] Step 9:

[1414] The server runs an SQL query against the database to obtain detailed information such as the product name and model number of the relevant part.

[1415] Step 10:

[1416] The server generates the matching results and simultaneously runs an emotion engine that extracts emotion information from the user's facial image, voice, etc.

[1417] Step 11:

[1418] The emotion engine (in the server) recognizes the user's emotions (e.g., stress, anxiety, joy, etc.) and generates the analysis results.

[1419] Step 12:

[1420] The server generates an adjusted matching result based on the analysis results of the emotion engine and sends it to the user's device, which also includes a feedback message according to the user's emotion.

[1421] Step 13:

[1422] The device receives the response from the server and decodes the JSON data.

[1423] Step 14:

[1424] The terminal displays the recognition result (e.g., "This is an antenna with model number XX") to the user, as well as a feedback message according to the user's emotion (e.g., "Good job, you're almost done").

[1425] Step 15:

[1426] The user checks the recognition results and feedback messages displayed on the terminal, and if necessary, proceeds with further investigation or response.

[1427] This specific step allows the system to help users quickly identify components in the field, and also recognizes the user's emotions and provides appropriate feedback.

[1428] Example 2

[1429] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1430] In base station construction work, there is a need for a method that allows even new employees or newly transferred employees to quickly and accurately identify components. There is also a need for a feedback function to reduce the stress and anxiety of users who work long hours. There has not been a system that meets these needs, so there is a need to solve these issues.

[1431] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1432] In this invention, the server includes means for receiving an image taken by a user, means for analyzing the image and extracting feature information, means for comparing the extracted feature information with information in a database, means for recognizing the user's emotion, means for generating feedback based on the emotion recognition result, and means for transmitting the generated feedback to the user's terminal. This allows the user to easily identify components and receive appropriate feedback regarding changes in emotion during the process, thereby improving work efficiency and the user experience.

[1433] The "means for receiving images taken by a user" refers to a device or method for acquiring image data taken by a user with a photographing device such as a smartphone through communication.

[1434] "Means for analyzing images and extracting feature information" refers to a device or method that analyzes captured image data using machine learning algorithms or image processing techniques to extract features such as shape, text, and color within the image.

[1435] The "means for matching with information in a database" refers to a device or method that searches for items that match information in a pre-registered database based on the extracted characteristic information and generates a matching result.

[1436] "Means for recognizing user emotions" refers to a device or method for analyzing a user's facial expressions and tone of voice from image or audio data, etc., to detect the user's emotional state (e.g., stress, anxiety, joy, etc.).

[1437] The "means for generating feedback based on emotion recognition results" refers to a device or method for creating an appropriate feedback message according to the recognized emotional state of the user and notifying the user of the appropriate feedback message.

[1438] The "means for transmitting the generated feedback to the user's terminal" refers to a device or method for transmitting the generated feedback message from the server to the user's terminal such as a smartphone or tablet via data communication.

[1439] This invention is a system for quickly and accurately identifying components in base station construction work. It aims to improve the user experience by recognizing the user's emotions and providing feedback based on those emotions. This allows even new employees and newly transferred employees to contribute immediately to the work. The system receives images taken by the user, analyzes the images to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback based on the results.

[1440] The specific configuration of the system is as follows.

[1441] 1. Receiving images

[1442] The user uses the smartphone camera to take an image of the component at the base station site. The captured image is then sent to the system using a communication application (e.g., LINE or WhatsApp). The device receives the image via the communication application and temporarily stores the image data in its internal memory. At this time, the device converts the image into an appropriate format (e.g., JPEG or PNG).

[1443] 2. Sending images and preparing for analysis

[1444] The device generates a request to send the stored image data to the server and sends it to the server. The server then passes the received image data to the AI ​​module, which then begins image analysis. This AI module performs image analysis using a deep learning framework (e.g., TensorFlow, PyTorch).

[1445] 3. Extraction of feature information

[1446] The server uses an AI module to extract feature information such as shape, text, and color from the image, which is then used for database matching.

[1447] 4. Database Matching

[1448] The server compares the extracted feature information with the registered information in the database. The database is built using a common RDBMS (e.g., MySQL or PostgreSQL) and stores detailed information such as the product name and model number of the component. The server selects the most matching result and temporarily stores it.

[1449] 5. Performing Emotion Recognition

[1450] In parallel, the emotion engine extracts emotional information from the user's facial image, voice, and other data. The emotion engine performs emotion analysis using, for example, OpenCV or a natural language processing (NLP) model. The recognized emotional data (e.g., stress level, anxiety level, happiness level) is saved.

[1451] 6. Producing adjusted matching results

[1452] The server adds a feedback message to the matching result based on the emotion recognition results. For example, if the user is feeling stressed, it generates a message such as "Good job, you're almost done." This data is converted into JSON format and sent to the user's device.

[1453] 7. Viewing Results and Feedback

[1454] The device receives the response from the server and displays the analysis results and feedback on the user interface. The user can check the recognition result (e.g., "This is an antenna with model number XX") and the emotional feedback message on the device.

[1455] Specific examples

[1456] A user takes a picture of the newly installed antenna of the base station with their smartphone.

[1457] The user uses a communication application to send the image to the system.

[1458] The device receives the image, converts it into the appropriate format, and sends it to the server.

[1459] The server receives the images and analyzes them using an AI module.

[1460] The AI ​​module extracts characteristic information and compares it with a database.

[1461] The server identifies the matching component information, and in parallel, the emotion engine recognizes the user's emotion.

[1462] The server generates adjusted matching results and emotional feedback and sends them to the device.

[1463] The device will display the results, along with a message such as "This is an antenna with model number XX" and "Thank you for your hard work, it's almost done."

[1464] Prompt Sentence Examples

[1465] "I would like to develop a system that analyzes and identifies images of components taken at the base station's work site and returns the results. It also needs a function that recognizes the user's emotions and provides feedback. Please explain in detail the program flow for such a system."

[1466] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1467] Step 1:

[1468] The user uses the smartphone camera to take pictures of components (e.g., antennas, radios, and concrete poles) at the base station site. The input is the image taken by the user with the camera, and the output is an image file saved on the smartphone. Specifically, the user launches the smartphone's camera app, frames the component, and presses the capture button.

[1469] Step 2:

[1470] The user sends the captured image to the system using a communication application (e.g., LINE or WhatsApp). The input is an image file stored on the smartphone, and the output is image data sent through the communication application. Specifically, the user opens the communication application, displays the chat screen, attaches the image, and presses the send button.

[1471] Step 3:

[1472] The device receives images via a communication application and temporarily stores the image data in its internal memory. The input is the image data received from the communication application, and the output is an image file stored in a temporary storage folder. Specifically, the device receives an application notification, downloads the image data, and stores it in a temporary storage folder (e.g., / tmp / images).

[1473] Step 4:

[1474] The device converts the received image into an appropriate format (e.g., JPEG, PNG) and generates a request to send to the server. The input is the image file saved in the temporary storage folder, and the output is the image data sent to the server as an HTTP POST request. Specifically, the device checks the image format, performs conversion processing if necessary, and generates an HTTP POST request.

[1475] Step 5:

[1476] The server passes the received image data to the AI ​​module and begins image analysis. The input is the image data received as an HTTP POST request, and the output is the image data waiting to be analyzed. Specifically, the server obtains the image data via the file reception API and passes it to the AI ​​module.

[1477] Step 6:

[1478] The server uses an AI module to extract feature information such as shape, text, and color from images. The input is the image data passed to the AI ​​module, and the output is the extracted feature information. Specifically, the server runs an AI module (e.g., TensorFlow, PyTorch) and applies an image analysis algorithm to extract feature information.

[1479] Step 7:

[1480] The server compares the extracted feature information with the component information in the database. The input is the extracted feature information, and the output is the component information as a comparison result. Specifically, the server generates a database query and searches a database such as MySQL or PostgreSQL to obtain the most matching component information.

[1481] Step 8:

[1482] The server also uses an emotion engine to recognize the user's emotions. The input is the user's facial image and voice data, and the output is the recognized emotion data. Specifically, the server runs the emotion engine (e.g., OpenCV, NLP model) to extract emotion information from the facial image and voice.

[1483] Step 9:

[1484] The server generates an adjusted matching result and a feedback message based on the emotion recognition result. The input is the component information as the matching result and the recognized emotion data, and the output is the adjusted matching result and a feedback message. Specifically, the server creates a feedback message based on the emotion data and converts it into JSON format.

[1485] Step 10:

[1486] The server sends the generated feedback message to the user's device. The input is the matching result and feedback message converted to JSON format, and the output is the data sent to the user's device as an HTTP response. Specifically, the server sends the JSON data through an API endpoint.

[1487] Step 11:

[1488] The terminal receives the response from the server and displays the analysis results and feedback on the user interface. The input is the JSON data received from the server, and the output is the recognition results and feedback messages displayed on the user interface. Specifically, the terminal parses the JSON data and displays it in a format that is easy for the user to understand.

[1489] Specific examples

[1490] A user takes a picture of the newly installed antenna of the base station with their smartphone.

[1491] The user uses a communication application to send the image to the system.

[1492] The device receives the image, converts it into the appropriate format, and sends it to the server.

[1493] The server receives the images and analyzes them using an AI module.

[1494] The AI ​​module extracts characteristic information and compares it with a database.

[1495] The server identifies the matching component information, and in parallel, the emotion engine recognizes the user's emotion.

[1496] The server generates adjusted matching results and emotional feedback and sends them to the device.

[1497] The device will display the results, along with a message such as "This is an antenna with model number XX" and "Thank you for your hard work, it's almost done."

[1498] (Application example 2)

[1499] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1500] Conventional parts identification work in factories often relies on the experience and knowledge of the worker, making it difficult for new employees or newly transferred workers to quickly and accurately identify parts. Furthermore, there was no mechanism to reduce stress and anxiety during work, leading to calls for improvements to the work environment. To solve these issues, a system is needed that automates the parts identification process and improves work efficiency by providing feedback on the user's emotions.

[1501] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving an image taken by a user, means for analyzing the image and extracting feature information, means for comparing the extracted feature information with information in a database, means for analyzing the user's emotions using an emotion engine, and means for generating a feedback message based on the analyzed emotion information. This allows factory workers to easily identify components, and by receiving feedback that takes into account their emotions during work, it is possible to reduce stress and improve work efficiency.

[1502] "User" refers to the factory worker who takes and sends images to the system for part identification and emotion recognition.

[1503] "Means for receiving images" refers to a communication function for allowing the server to receive images taken by the user.

[1504] The "means for analyzing an image and extracting characteristic information" refers to an image analysis function for extracting characteristic information such as the shape, text, and color of a component from a received image.

[1505] "Database" refers to data storage for storing component information and feature data.

[1506] "Means for matching" refers to a computer program for comparing extracted characteristic information with information in a database and identifying matching components.

[1507] "User's device" refers to the device used to receive and display matching results and feedback messages, specifically a smartphone, tablet, or PC.

[1508] An "emotion engine" refers to an algorithm or software that analyzes facial images and voice data taken by a user to generate emotional information.

[1509] "Emotion information" is data that indicates the user's emotional state, and includes emotions such as stress, anxiety, and joy.

[1510] A "feedback message" refers to a response message provided to a user based on emotional information.

[1511] "System" refers to the collection of devices and software that integrates and operates all of these means.

[1512] The present invention is a system for improving the efficiency of component identification work in a factory and for improving the work environment by providing feedback on the emotions of workers. Specific embodiments for carrying out the invention are described below.

[1513] System Overview

[1514] The system receives images taken by the user, analyzes them to extract feature information, compares them with a database, and then uses an emotion engine to analyze the user's emotions and provides feedback based on the results. This allows workers to easily identify parts and receive feedback based on their emotions.

[1515] Hardware and software used

[1516] Hardware

[1517] Smartphone terminal: Used by factory workers to take and send images of parts.

[1518] Robot: Used to move autonomously within the factory and take images of components.

[1519] Server: Performs image analysis, database matching, emotion recognition, and feedback message generation.

[1520] software

[1521] Image analysis: Extracting feature information from images using TensorFlow and OpenCV.

[1522] Database: MySQL is used to store and manage component information.

[1523] Emotion Recognition: Analyze user emotions using Azure Cognitive Services and Google Cloud Natural Language API.

[1524] Communication applications: Use common communication applications (e.g., LINE or WhatsApp) to send images.

[1525] System operation explanation

[1526] Taking and sending images

[1527] The user takes a picture of the component using a smartphone or a camera mounted on the robot and sends it to the server via a communication application.

[1528] Image analysis and feature information extraction

[1529] The server analyzes the received images using TensorFlow and OpenCV to extract feature information such as shape, text, and color, allowing it to compare detailed information about the identified components with a database.

[1530] Emotion Recognition and Feedback

[1531] In parallel, the server analyzes the user's facial image and voice data using Azure Cognitive Services and Google Cloud Natural Language API to generate emotional information, and then generates a feedback message based on the results and sends it to the user's device.

[1532] Specific examples

[1533] For example, suppose a user takes a picture of a component (e.g., a screw) in a factory using their smartphone. The image is sent to a server via a communication application (e.g., LINE or WhatsApp). The server analyzes the image, extracts the component's characteristics, and compares them with a database. The emotion engine then analyzes the user's emotions and generates a feedback message such as "This screw is model number XX" or "Thank you for your hard work. The work is going well," which is then sent to the user's device.

[1534] Examples of prompt statements

[1535] image_path = 'path_to_image'

[1536] user_data = 'Information including voice data and user facial images'

[1537] result = main(image_path, user_data)

[1538] print(json.dumps(result, indent=4, ensure_ascii=False))

[1539] In this way, the present invention allows factory workers to quickly and accurately identify parts and contributes to improving the work environment by providing feedback on their emotions while working.

[1540] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1541] Step 1:

[1542] The user takes an image of the part. Using a smartphone or a camera mounted on a robot, the image of the part is captured. This generates image data, which then becomes the input for processing in the next step.

[1543] Step 2:

[1544] The user sends the image they have taken to their device via a communication application, such as LINE or WhatsApp. During this process, the image data is sent from the smartphone to the server.

[1545] Step 3:

[1546] The server temporarily stores the images received from the user. The image data is converted into an appropriate format (JPEG, PNG, etc.) and stored in the internal memory. This stored image data is used in the next analysis step.

[1547] Step 4:

[1548] The server analyzes the stored image data using TensorFlow and OpenCV. Image analysis extracts feature information such as shape, text, and color. This feature information becomes input data for matching with the database.

[1549] Step 5:

[1550] The server compares the extracted feature information with the component information in the database, which stores information such as component model numbers and names. As a result of the comparison, the component information that best matches the information is identified.

[1551] Step 6:

[1552] The server simultaneously collects the user's emotional information and analyzes it using Azure Cognitive Services and Google Cloud Natural Language API. Facial images and voice data are input as the user's emotional data, and this emotional data is analyzed to output emotional information such as stress, anxiety, and joy.

[1553] Step 7:

[1554] The server generates a feedback message based on the analysis results of the emotion engine. An appropriate feedback message is generated based on the component information of the collation result and the user's emotion information. This feedback message is then sent to the user terminal.

[1555] Step 8:

[1556] The terminal displays the matching results and feedback messages received from the server to the user. The user can check information on the terminal, such as "This screw is model number XX" or "Thank you for your hard work, the work is going well." This improves work efficiency and the user experience.

[1557] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1558] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1559] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1560] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1561] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1562] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1563] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1564] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1565] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1566] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1567] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1568] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1569] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1570] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1571] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1572] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1573] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1574] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1575] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1576] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1577] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1578] The following is further disclosed regarding the above embodiment.

[1579] (Claim 1)

[1580] means for receiving an image captured by a user;

[1581] A means for analyzing an image and extracting feature information;

[1582] A means for comparing the extracted feature information with information in a database;

[1583] means for transmitting the matching result to a user's terminal;

[1584] A system including:

[1585] (Claim 2)

[1586] 10. The system of claim 1, further comprising a communication application for transmitting images taken by a user.

[1587] (Claim 3)

[1588] 2. The system according to claim 1, further comprising a means for displaying detailed information about the component based on the characteristic information.

[1589] (Claim 4)

[1590] 2. The system of claim 1, further comprising means for using an AI module when matching the extracted feature information with information in the database.

[1591] (Claim 5)

[1592] 2. The system according to claim 1, further comprising an interface for displaying the matching results on a user's terminal.

[1593] "Example 1"

[1594] (Claim 1)

[1595] a device for receiving images captured by a user;

[1596] an analysis device that analyzes the image and extracts feature information;

[1597] a matching device that matches the extracted feature information with information in a database;

[1598] a communication device that transmits the matching result to a user device;

[1599] A system including:

[1600] (Claim 2)

[1601] 10. The system of claim 1, further comprising communication software for transmitting images captured by a user.

[1602] (Claim 3)

[1603] 2. The system according to claim 1, further comprising a display device that displays detailed information about the component based on the characteristic information.

[1604] "Application Example 1"

[1605] (Claim 1)

[1606] means for receiving an image captured by a user;

[1607] A means for analyzing an image and extracting feature information;

[1608] A means for comparing the extracted feature information with information in a database;

[1609] means for transmitting the matching result to a user's terminal;

[1610] means for the robot to take an image of the component;

[1611] A means for storing image data in a built-in computer, converting the format, and transmitting the data to an analysis module;

[1612] A means for extracting feature information such as shape, text, and color using an image analysis module;

[1613] a means for reporting information about the component to a management system using the extracted characteristic information;

[1614] A system including:

[1615] (Claim 2)

[1616] 10. The system of claim 1, further comprising a communication application for transmitting images taken by a user.

[1617] (Claim 3)

[1618] 2. The system according to claim 1, further comprising a means for displaying detailed information about the component based on the characteristic information.

[1619] "Example 2: Combining Emotion Engines"

[1620] (Claim 1)

[1621] means for receiving an image captured by a user;

[1622] A means for analyzing an image and extracting feature information;

[1623] A means for comparing the extracted feature information with information in a database;

[1624] means for recognizing a user's emotion;

[1625] a means for generating feedback based on the emotion recognition result;

[1626] means for transmitting the generated feedback to a user terminal;

[1627] A system including:

[1628] (Claim 2)

[1629] 10. The system of claim 1, further comprising a communication application for transmitting images taken by a user.

[1630] (Claim 3)

[1631] a means for displaying detailed information about the component based on the characteristic information;

[1632] 2. The system according to claim 1, further comprising means for displaying a feedback message according to the user's emotion.

[1633] "Application example 2 when combining emotion engines"

[1634] (Claim 1)

[1635] means for receiving an image captured by a user;

[1636] A means for analyzing an image and extracting feature information;

[1637] A means for comparing the extracted feature information with information in a database;

[1638] means for transmitting the matching result to a user's terminal;

[1639] means for analyzing user emotions using an emotion engine;

[1640] means for generating a feedback message based on the analyzed emotional information;

[1641] A system including:

[1642] (Claim 2)

[1643] 10. The system of claim 1, further comprising a communication application for transmitting images taken by a user.

[1644] (Claim 3)

[1645] 2. The system according to claim 1, further comprising a means for displaying detailed information about the component based on the characteristic information. [Explanation of symbols]

[1646] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving an image captured by a user; means for analyzing an image and extracting feature information; A means for comparing the extracted feature information with information in a database; means for transmitting the matching result to a user's terminal; A system including:

2. 10. The system of claim 1, further comprising a communication application for transmitting images taken by a user.

3. 2. The system according to claim 1, further comprising means for displaying detailed information about the component based on the characteristic information.

4. 2. The system according to claim 1, further comprising means for using an AI module when comparing the extracted feature information with information in the database.

5. 2. The system according to claim 1, further comprising an interface for displaying the collation results on a user's terminal.

Images