System

Abstract

A system is provided.SOLUTION: A system comprising: means for obtaining image data; means for processing the image data; means for extracting features from the processed image data; means for identifying a product name based on the extracted features; means for obtaining the identified product name from a database; and means for displaying the obtained product name and related information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In base station construction work, it is difficult for new employees and those transferred to other companies to quickly and accurately determine the types and installation status of a wide variety of components using only images. Due to lack of experience, it takes time to identify components, which reduces work efficiency. It is necessary to solve this problem and train employees who can immediately contribute to the company. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides a system including: means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name from a database, and means for displaying the acquired product name and related information. This allows even inexperienced personnel to quickly identify components and efficiently carry out their work.

[0006] "Image data" is a collection of visual information captured using a digital camera, smartphone, etc., and is data composed of pixels.

[0007] "Processing means" refers to software or hardware for analyzing and converting image data, and is a device or program that performs operations including preprocessing and feature extraction.

[0008] "Preprocessing" refers to early-stage processing that includes image data resizing, noise removal, color space conversion, etc., and is a preparatory process for improving the accuracy of image analysis.

[0009] "Features" are characteristics such as patterns, shapes, colors, and edges extracted from image data, and are data points used for image recognition.

[0010] An "image recognition model" is a computational model that uses machine learning and deep learning algorithms to analyze the features of image data and perform identification and classification based on that analysis.

[0011] "Product name" is the name of various components such as radios, antennas, and concrete poles, and is information that is uniquely identified in the database.

[0012] A "database" is a collection of data organized in a specific structure, a system that stores product names and related information and enables search and retrieval operations.

[0013] The "display means" is a device or program for visualizing information on the display of a terminal, and is a display method including a user interface.

[0014] A "server" is a computer system that processes data and provides information in response to requests from client terminals.

[0015] A "terminal" is a device that is directly operated by a user and provides an interface for image capture, data display, etc., and includes devices such as smartphones and tablets.

[0016] A "user" is a human operator who captures image data and operates the system to identify components. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] In order to implement this invention, the following specific steps must be taken.

[0039] System configuration

[0040] This system is mainly composed of three components: a server, a terminal, and a user. The server provides a back-end environment for advanced analysis, while the terminal functions as an interface with the user. The user operates the terminal to collect image data and identifies components through the system.

[0041] Program processing flow

[0042] The specific flow of program processing in this system will be explained below.

[0043] 1. Taking and uploading images

[0044] The user uses the device's camera to take photos of components such as radios, antennas, and concrete poles. The captured images are selected within the device's application and sent to the server using the "Image Upload" function. The device then preprocesses the image data (resizing, noise removal, etc.) before preparing it for transfer to the server.

[0045] As a specific example, a user can take a picture of a base station antenna and tap the "upload image" button, which sends the image to the server.

[0046] 2. Receiving and preprocessing image data

[0047] The server receives the image data sent from the device. This image data undergoes further preprocessing (e.g., color space conversion, edge detection) to prepare it for more accurate analysis. Once preprocessed, the image data is used in the next step.

[0048] 3. Analysis using image recognition models

[0049] The preprocessed image data is input into the server's image recognition model, which uses machine learning algorithms such as deep learning to extract features from the image. Based on the extracted features, the server identifies the most similar product name. This identified product name is used in the next step.

[0050] As a specific example, the server analyzes an uploaded image of an antenna and identifies that the image is an antenna of "Model A123."

[0051] 4. Obtaining product name and related information

[0052] The server uses the identified product name to retrieve product information (product name, specifications, installation method, etc.) from the database. This information is very important for users to quickly obtain the information they need.

[0053] 5. Returning and displaying results

[0054] The server encodes the acquired product information in JSON format and sends it back to the device. The device receives this response, decodes it, and displays it in the user interface. The user can check the displayed information and take further action as necessary.

[0055] As a concrete example, the server encodes the antenna information for "Model A123" in JSON format and sends it back to the device. The device receives this and displays "This antenna is model A123. The specifications are as follows: ..."

[0056] User Feedback

[0057] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server, which receives and analyzes this feedback to help improve the system.

[0058] Specifically, users tap the "Feedback" button, write down the problems and suggestions for fixes, and submit them. The server analyzes this information and uses it to retrain the system and update the database.

[0059] In this way, the present invention improves work efficiency by quickly and accurately identifying base station components using image data. Even inexperienced newcomers and transferred employees can immediately contribute, and the overall work flow proceeds smoothly.

[0060] The above is a specific embodiment for carrying out the invention.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data on the device.

[0064] Step 2:

[0065] The device receives the selected image data and performs preprocessing, such as resizing the image, removing noise, and converting the color space, to prepare it for analysis. This preprocessing improves the accuracy of the analysis on the server.

[0066] Step 3:

[0067] The device sends the preprocessed image data to the server as an HTTP POST request to the server's API endpoint, where the image data is encoded and transferred.

[0068] Step 4:

[0069] The server receives the image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to prepare the image data in a state suitable for analysis.

[0070] Step 5:

[0071] The server inputs the preprocessed image data into an image recognition model, which uses a deep learning algorithm to extract features from the image and identify similar product names based on the extracted features.

[0072] Step 6:

[0073] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.), thereby collecting all the information the user needs.

[0074] Step 7:

[0075] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response, which prepares the terminal to receive data from the server.

[0076] Step 8:

[0077] The terminal receives and decodes the JSON data returned by the server. The decoded data is displayed in the user interface within the application, allowing the user to check the displayed product information.

[0078] Step 9:

[0079] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Specifically, the user taps the "Feedback" button, enters the problem and a correction suggestion, and submits it.

[0080] Step 10:

[0081] The server receives and analyzes the feedback data sent by users. Based on the analysis results, it retrains the system and image recognition model, updates the database, and aims to improve the accuracy of the entire system.

[0082] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes from capturing image data to analyzing, displaying the results, and collecting feedback.

[0083] Example 1

[0084] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] Conventional product identification systems using image data have the problem of being prone to misidentification due to low analysis accuracy on the server side, making it difficult to obtain accurate product information. Furthermore, the lack of a user feedback function makes it difficult to improve the system. Therefore, there is a need for the development of a system that can process image data quickly and accurately and provide product information to users.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0087] In this invention, the server includes means for preprocessing image data, means for transmitting the preprocessed image data to the server, means for further processing the image data received by the server, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name and related information from a database, means for encoding and transmitting the acquired product name and related information, means for decoding and displaying the transmitted information, and means for receiving and analyzing feedback from a user. This makes it possible to quickly and accurately identify products using image data and improve business efficiency.

[0088] "Image data" is data that represents visual information in a digital format.

[0089] "Preprocessing" refers to processes such as resizing, noise removal, and color space conversion that are carried out to make image data easier to analyze.

[0090] A "server" is a computer system that processes and stores data and communicates with clients.

[0091] A "terminal" is a device such as a computer or smartphone that is operated by a user.

[0092] A "feature" is information that is extracted from image data and represents a specific property or pattern.

[0093] "Product name" is a name used to identify a specific product.

[0094] A "database" is a system that allows for efficient management and retrieval of organized collections of data.

[0095] "Encoding" is the process of converting data into a certain format.

[0096] "Decoding" is the process of restoring encoded data to its original form.

[0097] "Feedback" is the act of sending a user's evaluation or opinion about a system.

[0098] To implement this invention, the following specific steps must be taken. First, the system is composed of three entities: a server, a terminal, and a user. The server provides a back-end environment for performing advanced analysis, and the terminal functions as an interface with the user. The user operates the terminal to collect image data and identifies components through the system.

[0099] System configuration

[0100] Specific examples of the hardware and software used in this system are shown below.

[0101] Server: AWS EC2 instance

[0102] Device: Smartphone or tablet

[0103] Preprocessing tool: OpenCV

[0104] Image recognition model: TensorFlow

[0105] Database: MySQL

[0106] JSON encoding: Jackson library (Java)

[0107] UI display: React Native application

[0108] Program processing

[0109] First, the user uses the device's camera app to take a photo of a radio, antenna, concrete pole, or other component. The user selects the image and taps the "Upload Image" button to send it to the server. The device then performs preprocessing such as resizing and noise removal on the image data before preparing it for transfer.

[0110] The server receives the image data sent from the device and performs further advanced preprocessing (e.g., color space conversion, edge detection). This processing improves the accuracy of image analysis. After preprocessing, the image data is input into the server's image recognition model.

[0111] The server's image recognition model uses a deep learning algorithm, specifically a Convolutional Neural Network (CNN), to extract features from images. Based on the extracted features, the server identifies the most similar product name.

[0112] Using the identified product name, the server retrieves related product information (such as name, specifications, and installation method) from the database. This information is encoded in JSON format and sent back to the terminal. The terminal decodes the received JSON data and displays it on the user interface. The user can check the displayed information and take further action as needed.

[0113] Furthermore, if a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request, which the server receives and analyzes to help improve the system.

[0114] Specific examples

[0115] As a specific example of how it works, a user takes a photo of a base station antenna and taps the "Upload Image" button, which sends the image to the server. The server analyzes the uploaded antenna image and identifies it as a "Model A123" antenna. It then encodes the "Model A123" antenna information it retrieves from the database in JSON format and returns it to the device. The device that receives this displays the message, "This antenna is Model A123. The specifications are as follows: ..."

[0116] Prompt Sentence Examples

[0117] The user taps the "Feedback" button and sends a comment saying, "The displayed information is incorrect. The actual model is B456." The server receives this, analyzes the feedback, and uses it for retraining and updating the database.

[0118] In this way, the present invention uses image data to quickly and accurately identify base station components, improving work efficiency and enabling even inexperienced new employees or transferred employees to immediately contribute to the company.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] Step 1:

[0121] The user launches the camera app on their device and takes a photo of a radio, antenna, concrete pole, or other component. The image is then saved in the app.

[0122] Input: Physical image of the part

[0123] Output: Digital image data stored on the device

[0124] Specific behavior:

[0125] The user opens the camera app on their smartphone and takes a picture of the antenna. The image is then saved on the device.

[0126] Step 2:

[0127] The user selects the image they have taken in the application and taps the "Upload Image" button. The device then performs preprocessing on the selected image, such as resizing and noise reduction.

[0128] Input: Digital image data stored on the device

[0129] Output: Pre-processed digital image data

[0130] Specific behavior:

[0131] The user selects an image in the app and taps the "Upload Image" button. The device resizes and denoises the image.

[0132] Step 3:

[0133] The terminal transmits the preprocessed image data to the server.

[0134] Input: Preprocessed digital image data

[0135] Output: Preprocessed image data sent to the server

[0136] Specific behavior:

[0137] The terminal transmits the preprocessed image data to the server.

[0138] Step 4:

[0139] The server receives the image data sent from the terminal, and then performs further preprocessing on this image data (color space conversion, edge detection).

[0140] Input: Preprocessed image data

[0141] Output: Image data processed with edge detection etc.

[0142] Specific behavior:

[0143] The server receives the image sent from the device, converts the color space from RGB to grayscale, and applies an edge detection algorithm to highlight the features of the components.

[0144] Step 5:

[0145] The server inputs the preprocessed image data into the image recognition model, which uses a deep learning algorithm (CNN) to extract features from the image.

[0146] Input: Preprocessed image data

[0147] Output: Features extracted from the recognition model

[0148] Specific behavior:

[0149] The server inputs the image data into the machine learning model, and the CNN extracts image features through filters.

[0150] Step 6:

[0151] The server identifies the most similar product name based on the extracted features.

[0152] Input: Extracted features

[0153] Output: Identified product name

[0154] Specific behavior:

[0155] Based on the extracted features, the machine learning model identifies the uploaded antenna as "Model A123."

[0156] Step 7:

[0157] The server queries the database using the identified product name to retrieve relevant product information (such as name, specifications, installation instructions, etc.).

[0158] Input: Identified product name

[0159] Output: Product information retrieved from the database

[0160] Specific behavior:

[0161] The server requests information from the MySQL database using the product name "Model A123", and the database returns the relevant data to the server.

[0162] Step 8:

[0163] The server encodes the acquired product information in JSON format and sends it to the terminal.

[0164] Input: Product information retrieved from the database

[0165] Output: JSON encoded data

[0166] Specific behavior:

[0167] The server converts the product information into JSON format and sends it to the terminal.

[0168] Step 9:

[0169] The terminal decodes the received JSON data and displays it in the user interface.

[0170] Input: JSON format data

[0171] Output: Product information displayed in the user interface

[0172] Specific behavior:

[0173] The device analyzes the received data and displays on the user interface, "This antenna is model A123. The specifications are as follows: ..."

[0174] Step 10:

[0175] If the user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server.

[0176] Input: User feedback

[0177] Output: Feedback data sent to the server

[0178] Specific behavior:

[0179] The user taps the "Feedback" button and submits the message stating, "The displayed information is incorrect. The actual model is B456."

[0180] In this way, the system uses image data to accurately identify products and provide information quickly.

[0181] (Application example 1)

[0182] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] In modern manufacturing, parts management and maintenance are important, but it is difficult to centrally manage and quickly identify a huge number of parts. To solve this problem, it is necessary to accurately and efficiently identify parts and obtain maintenance information while minimizing manual work.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0185] In this invention, the server includes means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying an item name based on the extracted features, means for acquiring the identified item name from a database, means for displaying the acquired item name and related information, means for acquiring and transmitting component data using an image acquisition device mounted on a factory machine, and means for analyzing the transmitted data and automatically returning the item name and related information, thereby enabling accurate and efficient acquisition of component identification and maintenance information.

[0186] "Image data" refers to visual information captured using a camera or other imaging device.

[0187] "Processing means" refers to algorithms or programs for analyzing or converting image data.

[0188] A "feature amount" is an important attribute or index for identifying a component extracted from image data.

[0189] "Item name" refers to the official name or identification label of the identified component.

[0190] A "database" is a storage system that stores product names and related information and allows them to be searched and retrieved.

[0191] "Related information" is detailed data such as specifications, installation methods, and maintenance information related to components.

[0192] The "means for displaying" refers to a display or interface that allows the user to visually recognize the acquired product name and related information.

[0193] "Factory machinery" refers to industrial equipment and robots used on production lines and for maintenance work.

[0194] "Image capture devices" refer to cameras and sensors mounted on factory machinery.

[0195] "Transmission means" refers to a communication function for transmitting image data and analysis results to a server.

[0196] The "analysis means" refers to an algorithm or program used on the server side to extract item names and features from image data.

[0197] The "automatic return method" is a communication function that sends the item name and related information generated based on the analysis results from the server to the user.

[0198] MODE FOR CARRYING OUT THE INVENTION

[0199] To implement the present invention, the following system configuration and processing are required.

[0200] System Configuration

[0201] The system is primarily composed of three components: a server, a terminal, and a user. The server provides a back-end environment for advanced analysis, while the terminal functions as an interface with the user. The user operates the terminal to collect image data of components and obtains the item names and related information identified through the system.

[0202] Program processing flow

[0203] 1. Acquiring and uploading images

[0204] A photo of the component is taken using the image capture device of the factory machine installed on the terminal. The captured image is sent to the server via the terminal's application. The image is then pre-processed, with resizing and noise removal.

[0205] 2. Receiving and analyzing image data

[0206] The server receives the image data sent from the device and performs a detailed analysis using an image recognition model and deep learning algorithm, which extracts features from the image and identifies the item name.

[0207] 3. Obtaining and displaying product names and related information

[0208] Based on the identified item name, the server retrieves relevant information from a database, which is then displayed in the user interface.

[0209] Hardware and Software

[0210] The system uses the following hardware and software:

[0211] Hardware: Camera (image capture device) mounted on factory machinery

[0212] software:

[0213] OpenCV: A library for image capture and preprocessing

[0214] requests: A library for sending HTTP requests

[0215] JSON: The format in which data received from the server is decoded and displayed.

[0216] Deep learning frameworks (e.g., TensorFlow, PyTorch): for feature extraction and running image recognition models

[0217] Specific examples

[0218] As a concrete example, consider a factory robot identifying a new concrete pole. The robot's camera takes a picture of the pole and uploads the image to a server through an application. The server analyzes the image and sends the identified item name and related information back to the robot, allowing it to execute the appropriate maintenance process.

[0219] Prompt Sentence Examples

[0220] "Please create a program for an application that analyzes images of concrete pillars taken by a factory robot with a camera and obtains the item name and specifications."

[0221] In this way, the entire system works together to execute a series of processes from acquiring image data to analyzing it and displaying the information, thereby streamlining component management and maintenance work within the factory.

[0222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0223] Program processing steps

[0224] Step 1:

[0225] The user takes a photo of the part using an image capture device of the factory machine mounted on the terminal.

[0226] Input: Image of a part taken by a factory machine camera

[0227] Specific operation: The user operates an application on the device and captures an image with the camera.

[0228] Step 2:

[0229] The captured images are pre-processed on the device, including resizing and noise removal.

[0230] Input: Raw image data captured

[0231] Specific operation: The device application uses OpenCV to resize the image and remove noise.

[0232] Step 3:

[0233] The pre-processed image data is uploaded from the terminal to a server.

[0234] Input: Preprocessed image data

[0235] Specific operation: The terminal uses the requests library to send image data to the server as a POST request.

[0236] Step 4:

[0237] The server again preprocesses the received image data and prepares it for analysis.

[0238] Input: Preprocessed image data received from the device

[0239] Specific operation: The server uses OpenCV to perform additional preprocessing such as color space conversion and edge detection.

[0240] Step 5:

[0241] The server analyzes the preprocessed images using a deep learning framework (e.g., TensorFlow, PyTorch) and extracts features.

[0242] Input: Image data preprocessed for analysis

[0243] Specific operation: The server uses the generative AI model to extract features from image data.

[0244] Step 6:

[0245] Based on the extracted features, the server identifies the item name.

[0246] Input: extracted features

[0247] Specific operation: The server compares the features with the database and identifies the most similar item name.

[0248] Step 7:

[0249] The server retrieves relevant information from a database based on the identified item name.

[0250] Input: Identified item name

[0251] Specific behavior: The server executes a database (e.g., SQL database) query to retrieve the item name and related information.

[0252] Step 8:

[0253] The server encodes the item name and related information in JSON format and returns it to the terminal.

[0254] Input: Item name and related information obtained

[0255] Specific operation: The server encodes the relevant information in JSON format and sends it to the terminal as an HTTP response.

[0256] Step 9:

[0257] The terminal decodes the JSON data received from the server and displays it in the user interface.

[0258] Input: JSON format data received from the server

[0259] Specific behavior: The device decodes the JSON and updates the application screen to show the item name and related information to the user.

[0260] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0261] To implement this invention, the following specific steps must be taken. The present invention provides a system that improves the efficiency of base station construction work by combining image data analysis and user emotion recognition. This system is mainly composed of a server, terminals, users, and an emotion engine.

[0262] System configuration

[0263] The system consists of the following elements:

[0264] 1. Server: Responsible for analyzing image data and processing emotion data.

[0265] 2. Terminal: Captures and preprocesses image data and provides an interface with the user.

[0266] 3. User: An operator who operates the system and provides image data.

[0267] 4. Emotion engine: An engine that recognizes the user's emotions and adjusts the system's behavior based on those emotions.

[0268] Program processing flow

[0269] The specific flow of program processing in this system will be explained below.

[0270] 1. Taking and uploading images

[0271] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data in the device.

[0272] As a specific example, a user takes a picture of a base station antenna and taps the "upload image" button.

[0273] 2. Image data preprocessing and transmission

[0274] The device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of analysis on the server. The preprocessed image data is then sent to the server as an HTTP POST request.

[0275] 3. Receiving and analyzing image data

[0276] The server receives the image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to make it suitable for analysis. The preprocessed image data is then input into an image recognition model. The image recognition model uses a deep learning algorithm to extract features from the image and identify similar product names based on the extracted features.

[0277] As a specific example, the server analyzes an uploaded image of an antenna and identifies that the image is an antenna of "Model A123."

[0278] 4. Obtaining product name and related information

[0279] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.) The retrieved product information provides all the information the user needs.

[0280] 5. Returning and displaying results

[0281] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response. The terminal receives this response, decodes it, and displays it on the user interface. The user can then check the displayed product information.

[0282] As a concrete example, the server encodes the antenna information for "Model A123" in JSON format and sends it back to the device. The device receives this and displays "This antenna is model A123. The specifications are as follows: ..."

[0283] 6. Emotion Recognition and Feedback

[0284] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions. Specifically, it changes the content displayed in the user interface or provides additional help information.

[0285] For example, if a user makes a face expressing dissatisfaction with a result, the emotion engine could recognize this and display more detailed information or additional support options.

[0286] 7. Feedback Collection and Analysis

[0287] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Based on the emotional data acquired by the emotion engine, the priority of the feedback content is determined and work is done to improve the system as necessary.

[0288] For example, a user can tap a "Feedback" button, write down the problem and suggest fixes, and the server will analyze the feedback to help retrain the system and update the database.

[0289] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[0290] The processing flow will be explained below.

[0291] Step 1:

[0292] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data on the device.

[0293] Step 2:

[0294] The device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of the analysis on the server.

[0295] Step 3:

[0296] The device sends the preprocessed image data to the server as an HTTP POST request to the server's API endpoint, where the image data is encoded and transferred.

[0297] Step 4:

[0298] The server receives the image data sent from the device, and then preprocesses the image again (for example, by edge detection or image normalization) to prepare it for analysis.

[0299] Step 5:

[0300] The server inputs the preprocessed image data into an image recognition model, which uses a deep learning algorithm to extract features from the image and identify the most similar product name based on the extracted features.

[0301] Step 6:

[0302] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.) This information provides all the information the user needs.

[0303] Step 7:

[0304] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response, which prepares the terminal to receive data from the server.

[0305] Step 8:

[0306] The terminal receives and decodes the JSON data returned by the server. The decoded data is displayed in the application's user interface, allowing the user to check the displayed product information.

[0307] Step 9:

[0308] The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions.

[0309] Step 10:

[0310] The server adjusts the user interface display based on the recognized emotion, such as by displaying a screen offering additional help information or detailed explanations if the user expresses dissatisfaction.

[0311] Step 11:

[0312] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Based on the emotional data acquired by the emotion engine, the priority of the feedback content is determined and work is done to improve the system as necessary.

[0313] Step 12:

[0314] The server receives and analyzes the feedback data sent by users. Based on the analysis results, it retrains the system and image recognition model, updates the database, and aims to improve the accuracy of the entire system.

[0315] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[0316] Example 2

[0317] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] In conventional systems, image data analysis and user emotion recognition are often processed separately, resulting in insufficient information integration, reduced analysis accuracy, and difficulty in effectively incorporating user feedback. Furthermore, while base station construction work requires the rapid identification of various components and the provision of related information, there has been a lack of systems that can address this need. The present invention aims to solve these problems by providing an efficient system that integrates image data analysis, user emotion recognition, and feedback collection and reflection.

[0319] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0320] In this invention, the server includes means for preprocessing the image data again and extracting features, means for identifying a product name based on the extracted features, and means for the server to search a database based on the identified product name and obtain corresponding product information. This allows for efficient integration of multiple processing steps, providing immediate feedback to the user, and enabling system adjustment based on the user's emotions.

[0321] "Means for acquiring image data" refers to a device or method for collecting visual information as digital data using an input device such as a camera or scanner.

[0322] "Preprocessing means" refers to a method or device that performs operations such as resizing, noise removal, and color space conversion on acquired image data before analysis, thereby preparing the data in a form suitable for analysis.

[0323] The "means for transmitting to a server" refers to a device or method for transferring image data processed by a terminal to a remote server via a network.

[0324] A "means for extracting features" is a method or device that identifies important patterns and attributes such as shape, color, and texture from image data and extracts them as quantities.

[0325] The "means for identifying product names" refers to a method or device that compares the extracted features with a known database and identifies the product or object that best matches.

[0326] "Means for searching a database and obtaining relevant product information" refers to a method or device for searching a database using a specified product name or other key and extracting related information (product name, specifications, installation method, etc.).

[0327] The "means for transmitting the acquired product name and related information from the server to the terminal" refers to a device or method for transferring the information searched and acquired by the server to the terminal via the network.

[0328] "Means for displaying on a user interface" refers to a device or method for visually displaying acquired information on a terminal display or screen.

[0329] "Means for analyzing a user's emotions and adjusting the system's behavior based on those emotions" refers to a method or device that analyzes a user's facial expressions and voice to identify their emotional state, and flexibly changes the system's processing, such as changing the user interface or providing additional information, depending on those emotions.

[0330] To implement this invention, the following specific steps must be taken. This system combines image data analysis and user emotion recognition to streamline specific tasks. Specifically, it includes a series of tasks from acquiring image data, analyzing it to identifying product names, providing related information based on that information, and collecting feedback based on the user's emotions. This system consists of the following main components:

[0331] System Components

[0332] 1. Server:

[0333] The server acts as the main processing unit, preprocessing image data, extracting features, identifying product names, acquiring related information, and analyzing user emotion data. The server achieves this processing by receiving and processing HTTP requests using Python's Flask and running deep learning models using TensorFlow.

[0334] 2. Terminal:

[0335] The device is responsible for acquiring and preprocessing image data, sending the data to the server, displaying the results obtained from the server, and collecting user emotion data. The device is equipped with a camera and microphone, and uses Python's OpenCV to perform preprocessing such as image resizing, noise reduction, and color space conversion.

[0336] 3. User:

[0337] The user operates the device to take and upload image data, check the displayed results, and provide necessary feedback. The user's emotional data is analyzed by the emotion engine, and the system's behavior is adjusted accordingly.

[0338] 4. Emotion Engine:

[0339] The emotion engine recognizes emotions from the user's facial expressions and voice and captures them as data. For example, a model using Python's OpenCV or TensorFlow analyzes facial expression data entered by the user through the device's camera and recognizes the emotion (satisfaction, dissatisfaction, surprise, etc.). This data is used to adjust the system's display content and operations according to the user's emotions.

[0340] Specific examples of operation

[0341] The following example shows the specific operation of the system.

[0342] Example 1: Taking and uploading image data

[0343] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component, selects the image in the application, and taps the "Upload Image" button. For example, the user takes a photo of a base station antenna and taps the "Upload Image" button.

[0344] Example 2: Preprocessing and sending image data

[0345] The device resizes the captured image data, removes noise, and converts the color space, and then sends the preprocessed image data to the server as an HTTP POST request.

[0346] Example 3: Receiving and analyzing image data

[0347] The server receives the image data sent from the device, performs preprocessing again (edge ​​detection and image normalization), and then uses a deep learning model to extract features from the image and identify the product name based on those features. For example, the server may identify an image of an antenna as "Model A123."

[0348] Example 4: Retrieving product names and related information

[0349] The server searches the database based on the identified product name and obtains the relevant product information (product name, specifications, installation method, etc.).

[0350] Example 5: Returning and displaying results

[0351] The server encodes the acquired product information in JSON format and sends it to the device as an HTTP response. The device receives this, decodes it, and displays it in the user interface. For example, it might say, "This antenna is model A123. The specifications are as follows: ..."

[0352] Example 6: Emotion Recognition and Feedback

[0353] The emotion engine analyzes the user's facial expressions and voice to recognize emotions, which the server receives and adjusts the system's behavior. For example, if the user makes a dissatisfied face, the emotion engine will recognize this and display additional information or support options.

[0354] Example 7: Feedback collection and analysis

[0355] If the displayed information is incorrect, the user can use the feedback feature to send a correction request to the server, which analyzes the feedback and uses it to retrain the system and update the database.

[0356] Prompt Sentence Examples

[0357] "This antenna is model A123. Its specifications are as follows: ..."

[0358] Please provide feedback:

[0359] "Analysis results from the emotion engine: The user's emotion was recognized as dissatisfied. Additional supporting information is displayed."

[0360] As described above, the present invention provides an integrated system that includes everything from analyzing image data to recognizing user emotions and reflecting feedback, thereby improving work efficiency and accuracy.

[0361] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0362] Step 1:

[0363] Image capture and upload

[0364] Input: User instruction to take an image

[0365] How it works: A user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other object. The user launches the application, selects the camera function, frames the object, and taps the capture button.

[0366] Output: Image data stored in the device

[0367] Step 2:

[0368] Select and upload an image

[0369] Input: Image data stored on the device

[0370] How it works: The user selects an image they have taken in the application and taps the "Upload Image" button. This action causes the device to temporarily store the selected image data and prepare it for transmission to the server.

[0371] Output: Image data ready to send to the server

[0372] Step 3:

[0373] Image data preprocessing

[0374] Input: Selected image data

[0375] What it does: The device preprocesses the image data, using Python's OpenCV to resize the image (e.g., change the resolution to 1024x768), denoise it (apply a Gaussian filter), and perform color space conversion (from RGB to grayscale).

[0376] Output: Preprocessed image data

[0377] Step 4:

[0378] Sending data to the server

[0379] Input: Preprocessed image data

[0380] Operation: The device sends the preprocessed image data to the server as an HTTP POST request, which includes a unique identifier (e.g., a UUID).

[0381] Output: Image data received by the server

[0382] Step 5:

[0383] Re-preprocessing of image data

[0384] Input: Image data received by the server

[0385] Operation: The server performs preprocessing on the received image data again, specifically edge detection (using the Canny algorithm) and image normalization (adjusting values ​​to the 0-1 range).

[0386] Output: Re-preprocessed image data

[0387] Step 6:

[0388] Image analysis and feature extraction

[0389] Input: Re-preprocessed image data

[0390] How it works: The server inputs image data into a deep learning model using TensorFlow and extracts features, such as shapes and patterns from the image using a convolutional neural network (CNN).

[0391] Output: Features extracted from the image

[0392] Step 7:

[0393] Identifying the product name

[0394] Input: extracted features

[0395] How it works: The server searches the database based on the extracted features to identify the best-matching product name. Specifically, it compares the features with known database entries and identifies the entries with the highest similarity.

[0396] Output: Identified product name

[0397] Step 8:

[0398] Obtaining product information

[0399] Input: Identified product name

[0400] Operation: The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.). The database search uses SQL.

[0401] Output: Retrieved product information

[0402] Step 9:

[0403] Encoding and sending the results

[0404] Input: Retrieved product information

[0405] Operation: The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response.

[0406] Output: Product information sent to the terminal

[0407] Step 10:

[0408] Displaying the results

[0409] Input: Product information sent from the server

[0410] Operation: The device decodes the received JSON data and displays it in the user interface. The user can view product information on the display. For example, "This antenna is model A123. Specifications are as follows: ..."

[0411] Output: Product information displayed in the user interface

[0412] Step 11:

[0413] Emotion recognition and feedback collection

[0414] Input: User's facial expression data and voice data

[0415] How it works: The emotion engine uses the device's camera and microphone to collect the user's facial expressions and voice. It uses Python's OpenCV and TensorFlow to analyze the facial expressions and voice in real time and recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives the emotion data and adjusts the system's display and operations based on the emotion.

[0416] Output: Recognized user emotion data and the system's display content adjusted accordingly

[0417] Step 12:

[0418] Analyzing feedback and improving the system

[0419] Input: User-provided feedback and emotional data

[0420] How it works: The user taps the "Feedback" button to report problems with the displayed information and submit suggestions for correction. The server uses the emotion data to prioritize the feedback and use it to retrain the system and update the database.

[0421] Output: Improved system and updated database

[0422] Through the above processing steps, the system efficiently and accurately analyzes image data, recognizes user emotions, and collects and reflects feedback.

[0423] (Application example 2)

[0424] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0425] In recent years, quality control on factory production lines has become increasingly important, but traditional quality inspections often rely on human labor, resulting in many issues in terms of efficiency and accuracy. Furthermore, there is also the problem that feedback based on the emotions and satisfaction of inspectors is not sufficiently provided, making it difficult to improve the work environment. In response to these issues, the challenge is to provide a more efficient and highly accurate quality inspection system by utilizing image data analysis and emotion recognition.

[0426] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name from a database, means for displaying the acquired product name and related information, and means for recognizing the user's emotions and adjusting the behavior of the system. This makes it possible to improve the efficiency and accuracy of quality inspections on production lines and provide feedback based on the emotions of workers.

[0427] "Image data" refers to visual information captured using a camera or other imaging device.

[0428] "Processing means" refers to a method or device that performs preprocessing such as resizing, noise removal, and color space conversion on the acquired image data to make it suitable for analysis.

[0429] "Features" are attributes such as specific patterns, shapes, colors, and textures extracted from image data, and are information that enables product identification and classification.

[0430] "Product name" refers to the product's unique name or model number that is searched for in the database based on the identified features.

[0431] A "database" is an information collection system that stores product names and other related information based on identified features and allows them to be searched and retrieved.

[0432] A "user interface" is a screen or operation panel that allows the user to exchange information with the system, and is capable of displaying and inputting information.

[0433] The "means for recognizing emotions" refers to a method or device for analyzing emotions from a user's facial expressions and voice and identifying their emotional state.

[0434] The "means for adjusting the system's behavior" refers to a method or device for dynamically changing the system's operation or display content based on the results of emotion recognition.

[0435] The following specific steps must be taken to implement this invention. The present invention provides a system for improving the efficiency of quality control on factory production lines, combining image data analysis with user emotion recognition. This system is primarily composed of a server, a terminal, a user, and an emotion engine.

[0436] First, the user takes a picture of the product using the device's camera. After taking the picture, the user selects the image in the application and taps the "Upload Image" button. This selects the image data in the device. For example, the user takes a picture of a product on a production line and taps the "Upload Image" button.

[0437] Next, the device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of analysis on the server. The preprocessed image data is then sent to the server as an HTTP POST request.

[0438] The server receives image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to make it suitable for analysis. The preprocessed image data is input into an image recognition model. The image recognition model uses a deep learning algorithm with TensorFlow to extract features from the image and identify similar product names based on the extracted features. As a specific example, the server analyzes an uploaded product image and identifies that the image is of product "model number 123."

[0439] Next, the server searches a database based on the identified product name and retrieves the corresponding product information (product name, specifications, installation method, etc.). The retrieved product information provides all the information the user needs. The server encodes the retrieved product information in JSON format and sends it to the terminal as an HTTP response. The terminal receives this response, decodes it, and displays it on the user interface. The user can check the displayed product information. As a specific example, the server encodes the retrieved product information for "model number 123" in JSON format and sends it back to the terminal. The terminal receives this and displays, "This product is model number 123. Its specifications are as follows: ..."

[0440] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions. Specifically, it can change the content displayed in the user interface or provide additional help information. For example, if the user makes a face expressing dissatisfaction with the results, the emotion engine could recognize this and display more detailed information or additional support options.

[0441] For example, the prompt for taking a picture is:

[0442] "Take a photo of the defect in the product and upload the image. We'll find out what the defect is."

[0443] In this way, the system of the present invention can improve the efficiency and accuracy of quality control on factory production lines through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[0444] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0445] Step 1:

[0446] A user takes a picture of a product using the device's camera. The input is a physical image of the product, and the output is digital image data stored on the device. Specifically, the user opens an application on the device, uses the camera function to take a picture of the product, and saves the image in the application.

[0447] Step 2:

[0448] The user selects an image in the application and taps the "Upload Image" button. The input is the image data saved in step 1, and the output is the unprocessed image data ready to be sent from the device application to the server. Specifically, the user selects an image file in the application and taps the "Upload Image" button.

[0449] Step 3:

[0450] The device receives the selected image data and performs preprocessing. The input is the image data selected by the user, and the output is the preprocessed image data. Specifically, the device performs image resizing, noise removal, color space conversion, etc., to prepare the image for analysis. OpenCV's resize function is used for resizing, and blur or GaussianBlur functions are used for noise removal.

[0451] Step 4:

[0452] The preprocessed image data is sent to the server as an HTTP POST request. The input is the preprocessed image data, and the output is the data to be sent to the server. In specific operations, the terminal generates an HTTP POST request and sends the request including the preprocessed image data to the server.

[0453] Step 5:

[0454] The server receives the image data sent from the device and preprocesses it again. The input is the preprocessed image data sent from the device, and the output is image data suitable for analysis. Specifically, the server performs edge detection and image normalization, preparing it for input into the TensorFlow deep learning model.

[0455] Step 6:

[0456] The server uses a deep learning model to extract features from images and identify product names based on the identified features. The input is preprocessed image data, and the output is the identified product name. Specifically, the server uses TensorFlow to analyze images, detect specific features, and identify product names based on them.

[0457] Step 7:

[0458] The server searches the database based on the identified product name and retrieves the corresponding product information. The input is the identified product name and the output is the retrieved product information. Specifically, the server executes a database query to retrieve information such as the name, specifications, and installation method of the corresponding product.

[0459] Step 8:

[0460] The product information acquired by the server is encoded in JSON format and sent to the terminal as an HTTP response. The input is the acquired product information, and the output is JSON format data. Specifically, the server converts the acquired product information into JSON and sends it to the terminal as an HTTP response.

[0461] Step 9:

[0462] The terminal receives the response from the server, decodes it, and displays it on the user interface. The input is the JSON data returned from the server, and the output is the product information displayed on the user interface. Specifically, the terminal parses the JSON data and displays the product information on the screen.

[0463] Step 10:

[0464] The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions. The input is the user's facial expressions and voice data, and the output is the recognized emotion data. Specifically, the device or server analyzes the user's emotions using Haar Cascade or other emotion recognition algorithms.

[0465] Step 11:

[0466] The server adjusts the system behavior based on the emotion recognition results. The input is the recognized emotion data, and the output is the adjusted system behavior or additional display information. Specific actions include changing the display content of the user interface or providing additional help information.

[0467] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0468] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0469] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0470] [Second embodiment]

[0471] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0472] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0473] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0474] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0475] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0476] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0477] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0478] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0479] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0480] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0481] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0482] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0483] In order to implement this invention, the following specific steps must be taken.

[0484] System configuration

[0485] This system is mainly composed of three components: a server, a terminal, and a user. The server provides a back-end environment for advanced analysis, while the terminal functions as an interface with the user. The user operates the terminal to collect image data and identifies components through the system.

[0486] Program processing flow

[0487] The specific flow of program processing in this system will be explained below.

[0488] 1. Taking and uploading images

[0489] The user uses the device's camera to take photos of components such as radios, antennas, and concrete poles. The captured images are selected within the device's application and sent to the server using the "Image Upload" function. The device then preprocesses the image data (resizing, noise removal, etc.) before preparing it for transfer to the server.

[0490] As a specific example, a user can take a picture of a base station antenna and tap the "upload image" button, which sends the image to the server.

[0491] 2. Receiving and preprocessing image data

[0492] The server receives the image data sent from the device. This image data undergoes further preprocessing (e.g., color space conversion, edge detection) to prepare it for more accurate analysis. Once preprocessed, the image data is used in the next step.

[0493] 3. Analysis using image recognition models

[0494] The preprocessed image data is input into the server's image recognition model, which uses machine learning algorithms such as deep learning to extract features from the image. Based on the extracted features, the server identifies the most similar product name. This identified product name is used in the next step.

[0495] As a specific example, the server analyzes an uploaded image of an antenna and identifies that the image is an antenna of "Model A123."

[0496] 4. Obtaining product name and related information

[0497] The server uses the identified product name to retrieve product information (product name, specifications, installation method, etc.) from the database. This information is very important for users to quickly obtain the information they need.

[0498] 5. Returning and displaying results

[0499] The server encodes the acquired product information in JSON format and sends it back to the device. The device receives this response, decodes it, and displays it in the user interface. The user can check the displayed information and take further action as necessary.

[0500] As a concrete example, the server encodes the antenna information for "Model A123" in JSON format and sends it back to the device. The device receives this and displays "This antenna is model A123. The specifications are as follows: ..."

[0501] User Feedback

[0502] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server, which receives and analyzes this feedback to help improve the system.

[0503] Specifically, users tap the "Feedback" button, write down the problems and suggestions for fixes, and submit them. The server analyzes this information and uses it to retrain the system and update the database.

[0504] In this way, the present invention improves work efficiency by quickly and accurately identifying base station components using image data. Even inexperienced newcomers and transferred employees can immediately contribute, and the overall work flow proceeds smoothly.

[0505] The above is a specific embodiment for carrying out the invention.

[0506] The processing flow will be explained below.

[0507] Step 1:

[0508] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data on the device.

[0509] Step 2:

[0510] The device receives the selected image data and performs preprocessing, such as resizing the image, removing noise, and converting the color space, to prepare it for analysis. This preprocessing improves the accuracy of the analysis on the server.

[0511] Step 3:

[0512] The device sends the preprocessed image data to the server as an HTTP POST request to the server's API endpoint, where the image data is encoded and transferred.

[0513] Step 4:

[0514] The server receives the image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to prepare the image data in a state suitable for analysis.

[0515] Step 5:

[0516] The server inputs the preprocessed image data into an image recognition model, which uses a deep learning algorithm to extract features from the image and identify similar product names based on the extracted features.

[0517] Step 6:

[0518] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.), thereby collecting all the information the user needs.

[0519] Step 7:

[0520] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response, which prepares the terminal to receive data from the server.

[0521] Step 8:

[0522] The terminal receives and decodes the JSON data returned by the server. The decoded data is displayed in the user interface within the application, allowing the user to check the displayed product information.

[0523] Step 9:

[0524] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Specifically, the user taps the "Feedback" button, enters the problem and a correction suggestion, and submits it.

[0525] Step 10:

[0526] The server receives and analyzes the feedback data sent by users. Based on the analysis results, it retrains the system and image recognition model, updates the database, and aims to improve the accuracy of the entire system.

[0527] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes from capturing image data to analyzing, displaying the results, and collecting feedback.

[0528] Example 1

[0529] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0530] Conventional product identification systems using image data have the problem of being prone to misidentification due to low analysis accuracy on the server side, making it difficult to obtain accurate product information. Furthermore, the lack of a user feedback function makes it difficult to improve the system. Therefore, there is a need for the development of a system that can process image data quickly and accurately and provide product information to users.

[0531] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0532] In this invention, the server includes means for preprocessing image data, means for transmitting the preprocessed image data to the server, means for further processing the image data received by the server, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name and related information from a database, means for encoding and transmitting the acquired product name and related information, means for decoding and displaying the transmitted information, and means for receiving and analyzing feedback from a user. This makes it possible to quickly and accurately identify products using image data and improve business efficiency.

[0533] "Image data" is data that represents visual information in a digital format.

[0534] "Preprocessing" refers to processes such as resizing, noise removal, and color space conversion that are carried out to make image data easier to analyze.

[0535] A "server" is a computer system that processes and stores data and communicates with clients.

[0536] A "terminal" is a device such as a computer or smartphone that is operated by a user.

[0537] A "feature" is information that is extracted from image data and represents a specific property or pattern.

[0538] "Product name" is a name used to identify a specific product.

[0539] A "database" is a system that allows for efficient management and retrieval of organized collections of data.

[0540] "Encoding" is the process of converting data into a certain format.

[0541] "Decoding" is the process of restoring encoded data to its original form.

[0542] "Feedback" is the act of sending a user's evaluation or opinion about a system.

[0543] To implement this invention, the following specific steps must be taken. First, the system is composed of three entities: a server, a terminal, and a user. The server provides a back-end environment for performing advanced analysis, and the terminal functions as an interface with the user. The user operates the terminal to collect image data and identifies components through the system.

[0544] System configuration

[0545] Specific examples of the hardware and software used in this system are shown below.

[0546] Server: AWS EC2 instance

[0547] Device: Smartphone or tablet

[0548] Preprocessing tool: OpenCV

[0549] Image recognition model: TensorFlow

[0550] Database: MySQL

[0551] JSON encoding: Jackson library (Java)

[0552] UI display: React Native application

[0553] Program processing

[0554] First, the user uses the device's camera app to take a photo of a radio, antenna, concrete pole, or other component. The user selects the image and taps the "Upload Image" button to send it to the server. The device then performs preprocessing such as resizing and noise removal on the image data before preparing it for transfer.

[0555] The server receives the image data sent from the device and performs further advanced preprocessing (e.g., color space conversion, edge detection). This processing improves the accuracy of image analysis. After preprocessing, the image data is input into the server's image recognition model.

[0556] The server's image recognition model uses a deep learning algorithm, specifically a Convolutional Neural Network (CNN), to extract features from images. Based on the extracted features, the server identifies the most similar product name.

[0557] Using the identified product name, the server retrieves related product information (such as name, specifications, and installation method) from the database. This information is encoded in JSON format and sent back to the terminal. The terminal decodes the received JSON data and displays it on the user interface. The user can check the displayed information and take further action as needed.

[0558] Furthermore, if a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request, which the server receives and analyzes to help improve the system.

[0559] Specific examples

[0560] As a specific example of how it works, a user takes a photo of a base station antenna and taps the "Upload Image" button, which sends the image to the server. The server analyzes the uploaded antenna image and identifies it as a "Model A123" antenna. It then encodes the "Model A123" antenna information it retrieves from the database in JSON format and returns it to the device. The device that receives this displays the message, "This antenna is Model A123. The specifications are as follows: ..."

[0561] Prompt Sentence Examples

[0562] The user taps the "Feedback" button and sends a comment saying, "The displayed information is incorrect. The actual model is B456." The server receives this, analyzes the feedback, and uses it for retraining and updating the database.

[0563] In this way, the present invention uses image data to quickly and accurately identify base station components, improving work efficiency and enabling even inexperienced new employees or transferred employees to immediately contribute to the company.

[0564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0565] Step 1:

[0566] The user launches the camera app on their device and takes a photo of a radio, antenna, concrete pole, or other component. The image is then saved in the app.

[0567] Input: Physical image of the part

[0568] Output: Digital image data stored on the device

[0569] Specific behavior:

[0570] The user opens the camera app on their smartphone and takes a picture of the antenna. The image is then saved on the device.

[0571] Step 2:

[0572] The user selects the image they have taken in the application and taps the "Upload Image" button. The device then performs preprocessing on the selected image, such as resizing and noise reduction.

[0573] Input: Digital image data stored on the device

[0574] Output: Pre-processed digital image data

[0575] Specific behavior:

[0576] The user selects an image in the app and taps the "Upload Image" button. The device resizes and denoises the image.

[0577] Step 3:

[0578] The terminal transmits the preprocessed image data to the server.

[0579] Input: Preprocessed digital image data

[0580] Output: Preprocessed image data sent to the server

[0581] Specific behavior:

[0582] The terminal transmits the preprocessed image data to the server.

[0583] Step 4:

[0584] The server receives the image data sent from the terminal, and then performs further preprocessing on this image data (color space conversion, edge detection).

[0585] Input: Preprocessed image data

[0586] Output: Image data processed with edge detection etc.

[0587] Specific behavior:

[0588] The server receives the image sent from the device, converts the color space from RGB to grayscale, and applies an edge detection algorithm to highlight the features of the components.

[0589] Step 5:

[0590] The server inputs the preprocessed image data into the image recognition model, which uses a deep learning algorithm (CNN) to extract features from the image.

[0591] Input: Preprocessed image data

[0592] Output: Features extracted from the recognition model

[0593] Specific behavior:

[0594] The server inputs the image data into the machine learning model, and the CNN extracts image features through filters.

[0595] Step 6:

[0596] The server identifies the most similar product name based on the extracted features.

[0597] Input: Extracted features

[0598] Output: Identified product name

[0599] Specific behavior:

[0600] Based on the extracted features, the machine learning model identifies the uploaded antenna as "Model A123."

[0601] Step 7:

[0602] The server queries the database using the identified product name to retrieve relevant product information (such as name, specifications, installation instructions, etc.).

[0603] Input: Identified product name

[0604] Output: Product information retrieved from the database

[0605] Specific behavior:

[0606] The server requests information from the MySQL database using the product name "Model A123", and the database returns the relevant data to the server.

[0607] Step 8:

[0608] The server encodes the acquired product information in JSON format and sends it to the terminal.

[0609] Input: Product information retrieved from the database

[0610] Output: JSON encoded data

[0611] Specific behavior:

[0612] The server converts the product information into JSON format and sends it to the terminal.

[0613] Step 9:

[0614] The terminal decodes the received JSON data and displays it in the user interface.

[0615] Input: JSON format data

[0616] Output: Product information displayed in the user interface

[0617] Specific behavior:

[0618] The device analyzes the received data and displays on the user interface, "This antenna is model A123. The specifications are as follows: ..."

[0619] Step 10:

[0620] If the user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server.

[0621] Input: User feedback

[0622] Output: Feedback data sent to the server

[0623] Specific behavior:

[0624] The user taps the "Feedback" button and submits the message stating, "The displayed information is incorrect. The actual model is B456."

[0625] In this way, the system uses image data to accurately identify products and provide information quickly.

[0626] (Application example 1)

[0627] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0628] In modern manufacturing, parts management and maintenance are important, but it is difficult to centrally manage and quickly identify a huge number of parts. To solve this problem, it is necessary to accurately and efficiently identify parts and obtain maintenance information while minimizing manual work.

[0629] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0630] In this invention, the server includes means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying an item name based on the extracted features, means for acquiring the identified item name from a database, means for displaying the acquired item name and related information, means for acquiring and transmitting component data using an image acquisition device mounted on a factory machine, and means for analyzing the transmitted data and automatically returning the item name and related information, thereby enabling accurate and efficient acquisition of component identification and maintenance information.

[0631] "Image data" refers to visual information captured using a camera or other imaging device.

[0632] "Processing means" refers to algorithms or programs for analyzing or converting image data.

[0633] A "feature amount" is an important attribute or index for identifying a component extracted from image data.

[0634] "Item name" refers to the official name or identification label of the identified component.

[0635] A "database" is a storage system that stores product names and related information and allows them to be searched and retrieved.

[0636] "Related information" is detailed data such as specifications, installation methods, and maintenance information related to components.

[0637] The "means for displaying" refers to a display or interface that allows the user to visually recognize the acquired product name and related information.

[0638] "Factory machinery" refers to industrial equipment and robots used on production lines and for maintenance work.

[0639] "Image capture devices" refer to cameras and sensors mounted on factory machinery.

[0640] "Transmission means" refers to a communication function for transmitting image data and analysis results to a server.

[0641] The "analysis means" refers to an algorithm or program used on the server side to extract item names and features from image data.

[0642] The "automatic return method" is a communication function that sends the item name and related information generated based on the analysis results from the server to the user.

[0643] MODE FOR CARRYING OUT THE INVENTION

[0644] To implement the present invention, the following system configuration and processing are required.

[0645] System Configuration

[0646] The system is primarily composed of three components: a server, a terminal, and a user. The server provides a back-end environment for advanced analysis, while the terminal functions as an interface with the user. The user operates the terminal to collect image data of components and obtains the item names and related information identified through the system.

[0647] Program processing flow

[0648] 1. Acquiring and uploading images

[0649] A photo of the component is taken using the image capture device of the factory machine installed on the terminal. The captured image is sent to the server via the terminal's application. The image is then pre-processed, with resizing and noise removal.

[0650] 2. Receiving and analyzing image data

[0651] The server receives the image data sent from the device and performs a detailed analysis using an image recognition model and deep learning algorithm, which extracts features from the image and identifies the item name.

[0652] 3. Obtaining and displaying product names and related information

[0653] Based on the identified item name, the server retrieves relevant information from a database, which is then displayed in the user interface.

[0654] Hardware and Software

[0655] The system uses the following hardware and software:

[0656] Hardware: Camera (image capture device) mounted on factory machinery

[0657] software:

[0658] OpenCV: A library for image capture and preprocessing

[0659] requests: A library for sending HTTP requests

[0660] JSON: The format in which data received from the server is decoded and displayed.

[0661] Deep learning frameworks (e.g., TensorFlow, PyTorch): for feature extraction and running image recognition models

[0662] Specific examples

[0663] As a concrete example, consider a factory robot identifying a new concrete pole. The robot's camera takes a picture of the pole and uploads the image to a server through an application. The server analyzes the image and sends the identified item name and related information back to the robot, allowing it to execute the appropriate maintenance process.

[0664] Prompt Sentence Examples

[0665] "Please create a program for an application that analyzes images of concrete pillars taken by a factory robot with a camera and obtains the item name and specifications."

[0666] In this way, the entire system works together to execute a series of processes from acquiring image data to analyzing it and displaying the information, thereby streamlining component management and maintenance work within the factory.

[0667] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0668] Program processing steps

[0669] Step 1:

[0670] The user takes a photo of the part using an image capture device of the factory machine mounted on the terminal.

[0671] Input: Image of a part taken by a factory machine camera

[0672] Specific operation: The user operates an application on the device and captures an image with the camera.

[0673] Step 2:

[0674] The captured images are pre-processed on the device, including resizing and noise removal.

[0675] Input: Raw image data captured

[0676] Specific operation: The device application uses OpenCV to resize the image and remove noise.

[0677] Step 3:

[0678] The pre-processed image data is uploaded from the terminal to a server.

[0679] Input: Preprocessed image data

[0680] Specific operation: The terminal uses the requests library to send image data to the server as a POST request.

[0681] Step 4:

[0682] The server again preprocesses the received image data and prepares it for analysis.

[0683] Input: Preprocessed image data received from the device

[0684] Specific operation: The server uses OpenCV to perform additional preprocessing such as color space conversion and edge detection.

[0685] Step 5:

[0686] The server analyzes the preprocessed images using a deep learning framework (e.g., TensorFlow, PyTorch) and extracts features.

[0687] Input: Image data preprocessed for analysis

[0688] Specific operation: The server uses the generative AI model to extract features from image data.

[0689] Step 6:

[0690] Based on the extracted features, the server identifies the item name.

[0691] Input: extracted features

[0692] Specific operation: The server compares the features with the database and identifies the most similar item name.

[0693] Step 7:

[0694] The server retrieves relevant information from a database based on the identified item name.

[0695] Input: Identified item name

[0696] Specific behavior: The server executes a database (e.g., SQL database) query to retrieve the item name and related information.

[0697] Step 8:

[0698] The server encodes the item name and related information in JSON format and returns it to the terminal.

[0699] Input: Item name and related information obtained

[0700] Specific operation: The server encodes the relevant information in JSON format and sends it to the terminal as an HTTP response.

[0701] Step 9:

[0702] The terminal decodes the JSON data received from the server and displays it in the user interface.

[0703] Input: JSON format data received from the server

[0704] Specific behavior: The device decodes the JSON and updates the application screen to show the item name and related information to the user.

[0705] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0706] To implement this invention, the following specific steps must be taken. The present invention provides a system that improves the efficiency of base station construction work by combining image data analysis and user emotion recognition. This system is mainly composed of a server, terminals, users, and an emotion engine.

[0707] System configuration

[0708] The system consists of the following elements:

[0709] 1. Server: Responsible for analyzing image data and processing emotion data.

[0710] 2. Terminal: Captures and preprocesses image data and provides an interface with the user.

[0711] 3. User: An operator who operates the system and provides image data.

[0712] 4. Emotion engine: An engine that recognizes the user's emotions and adjusts the system's behavior based on those emotions.

[0713] Program processing flow

[0714] The specific flow of program processing in this system will be explained below.

[0715] 1. Taking and uploading images

[0716] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data in the device.

[0717] As a specific example, a user takes a picture of a base station antenna and taps the "upload image" button.

[0718] 2. Image data preprocessing and transmission

[0719] The device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of analysis on the server. The preprocessed image data is then sent to the server as an HTTP POST request.

[0720] 3. Receiving and analyzing image data

[0721] The server receives the image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to make it suitable for analysis. The preprocessed image data is then input into an image recognition model. The image recognition model uses a deep learning algorithm to extract features from the image and identify similar product names based on the extracted features.

[0722] As a specific example, the server analyzes an uploaded image of an antenna and identifies that the image is an antenna of "Model A123."

[0723] 4. Obtaining product name and related information

[0724] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.) The retrieved product information provides all the information the user needs.

[0725] 5. Returning and displaying results

[0726] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response. The terminal receives this response, decodes it, and displays it on the user interface. The user can then check the displayed product information.

[0727] As a concrete example, the server encodes the antenna information for "Model A123" in JSON format and sends it back to the device. The device receives this and displays "This antenna is model A123. The specifications are as follows: ..."

[0728] 6. Emotion Recognition and Feedback

[0729] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions. Specifically, it changes the content displayed in the user interface or provides additional help information.

[0730] For example, if a user makes a face expressing dissatisfaction with a result, the emotion engine could recognize this and display more detailed information or additional support options.

[0731] 7. Feedback Collection and Analysis

[0732] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Based on the emotional data acquired by the emotion engine, the priority of the feedback content is determined and work is done to improve the system as necessary.

[0733] For example, a user can tap a "Feedback" button, write down the problem and suggest fixes, and the server will analyze the feedback to help retrain the system and update the database.

[0734] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[0735] The processing flow will be explained below.

[0736] Step 1:

[0737] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data on the device.

[0738] Step 2:

[0739] The device receives the selected image data and performs preprocessing, such as resizing the image, removing noise, and converting the color space, to prepare it for analysis. This preprocessing improves the accuracy of the analysis on the server.

[0740] Step 3:

[0741] The device sends the preprocessed image data to the server as an HTTP POST request to the server's API endpoint, where the image data is encoded and transferred.

[0742] Step 4:

[0743] The server receives the image data sent from the device, and then preprocesses the image again (for example, by edge detection or image normalization) to prepare it for analysis.

[0744] Step 5:

[0745] The server inputs the preprocessed image data into an image recognition model, which uses a deep learning algorithm to extract features from the image and identify the most similar product name based on the extracted features.

[0746] Step 6:

[0747] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.) This information provides all the information the user needs.

[0748] Step 7:

[0749] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response, which prepares the terminal to receive data from the server.

[0750] Step 8:

[0751] The terminal receives and decodes the JSON data returned by the server. The decoded data is displayed in the application's user interface, allowing the user to check the displayed product information.

[0752] Step 9:

[0753] The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions.

[0754] Step 10:

[0755] The server adjusts the user interface display based on the recognized emotion, such as by displaying a screen offering additional help information or detailed explanations if the user expresses dissatisfaction.

[0756] Step 11:

[0757] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Based on the emotional data acquired by the emotion engine, the priority of the feedback content is determined and work is done to improve the system as necessary.

[0758] Step 12:

[0759] The server receives and analyzes the feedback data sent by users. Based on the analysis results, it retrains the system and image recognition model, updates the database, and aims to improve the accuracy of the entire system.

[0760] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[0761] Example 2

[0762] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0763] In conventional systems, image data analysis and user emotion recognition are often processed separately, resulting in insufficient information integration, reduced analysis accuracy, and difficulty in effectively incorporating user feedback. Furthermore, while base station construction work requires the rapid identification of various components and the provision of related information, there has been a lack of systems that can address this need. The present invention aims to solve these problems by providing an efficient system that integrates image data analysis, user emotion recognition, and feedback collection and reflection.

[0764] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0765] In this invention, the server includes means for preprocessing the image data again and extracting features, means for identifying a product name based on the extracted features, and means for the server to search a database based on the identified product name and obtain corresponding product information. This allows for efficient integration of multiple processing steps, providing immediate feedback to the user, and enabling system adjustment based on the user's emotions.

[0766] "Means for acquiring image data" refers to a device or method for collecting visual information as digital data using an input device such as a camera or scanner.

[0767] "Preprocessing means" refers to a method or device that performs operations such as resizing, noise removal, and color space conversion on acquired image data before analysis, thereby preparing the data in a form suitable for analysis.

[0768] The "means for transmitting to a server" refers to a device or method for transferring image data processed by a terminal to a remote server via a network.

[0769] A "means for extracting features" is a method or device that identifies important patterns and attributes such as shape, color, and texture from image data and extracts them as quantities.

[0770] The "means for identifying product names" refers to a method or device that compares the extracted features with a known database and identifies the product or object that best matches.

[0771] "Means for searching a database and obtaining relevant product information" refers to a method or device for searching a database using a specified product name or other key and extracting related information (product name, specifications, installation method, etc.).

[0772] The "means for transmitting the acquired product name and related information from the server to the terminal" refers to a device or method for transferring the information searched and acquired by the server to the terminal via the network.

[0773] "Means for displaying on a user interface" refers to a device or method for visually displaying acquired information on a terminal display or screen.

[0774] "Means for analyzing a user's emotions and adjusting the system's behavior based on those emotions" refers to a method or device that analyzes a user's facial expressions and voice to identify their emotional state, and flexibly changes the system's processing, such as changing the user interface or providing additional information, depending on those emotions.

[0775] To implement this invention, the following specific steps must be taken. This system combines image data analysis and user emotion recognition to streamline specific tasks. Specifically, it includes a series of tasks from acquiring image data, analyzing it to identifying product names, providing related information based on that information, and collecting feedback based on the user's emotions. This system consists of the following main components:

[0776] System Components

[0777] 1. Server:

[0778] The server acts as the main processing unit, preprocessing image data, extracting features, identifying product names, acquiring related information, and analyzing user emotion data. The server achieves this processing by receiving and processing HTTP requests using Python's Flask and running deep learning models using TensorFlow.

[0779] 2. Terminal:

[0780] The device is responsible for acquiring and preprocessing image data, sending the data to the server, displaying the results obtained from the server, and collecting user emotion data. The device is equipped with a camera and microphone, and uses Python's OpenCV to perform preprocessing such as image resizing, noise reduction, and color space conversion.

[0781] 3. User:

[0782] The user operates the device to take and upload image data, check the displayed results, and provide necessary feedback. The user's emotional data is analyzed by the emotion engine, and the system's behavior is adjusted accordingly.

[0783] 4. Emotion Engine:

[0784] The emotion engine recognizes emotions from the user's facial expressions and voice and captures them as data. For example, a model using Python's OpenCV or TensorFlow analyzes facial expression data entered by the user through the device's camera and recognizes the emotion (satisfaction, dissatisfaction, surprise, etc.). This data is used to adjust the system's display content and operations according to the user's emotions.

[0785] Specific examples of operation

[0786] The following example shows the specific operation of the system.

[0787] Example 1: Taking and uploading image data

[0788] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component, selects the image in the application, and taps the "Upload Image" button. For example, the user takes a photo of a base station antenna and taps the "Upload Image" button.

[0789] Example 2: Preprocessing and sending image data

[0790] The device resizes the captured image data, removes noise, and converts the color space, and then sends the preprocessed image data to the server as an HTTP POST request.

[0791] Example 3: Receiving and analyzing image data

[0792] The server receives the image data sent from the device, performs preprocessing again (edge ​​detection and image normalization), and then uses a deep learning model to extract features from the image and identify the product name based on those features. For example, the server may identify an image of an antenna as "Model A123."

[0793] Example 4: Retrieving product names and related information

[0794] The server searches the database based on the identified product name and obtains the relevant product information (product name, specifications, installation method, etc.).

[0795] Example 5: Returning and displaying results

[0796] The server encodes the acquired product information in JSON format and sends it to the device as an HTTP response. The device receives this, decodes it, and displays it in the user interface. For example, it might say, "This antenna is model A123. The specifications are as follows: ..."

[0797] Example 6: Emotion Recognition and Feedback

[0798] The emotion engine analyzes the user's facial expressions and voice to recognize emotions, which the server receives and adjusts the system's behavior. For example, if the user makes a dissatisfied face, the emotion engine will recognize this and display additional information or support options.

[0799] Example 7: Feedback collection and analysis

[0800] If the displayed information is incorrect, the user can use the feedback feature to send a correction request to the server, which analyzes the feedback and uses it to retrain the system and update the database.

[0801] Prompt Sentence Examples

[0802] "This antenna is model A123. Its specifications are as follows: ..."

[0803] Please provide feedback:

[0804] "Analysis results from the emotion engine: The user's emotion was recognized as dissatisfied. Additional supporting information is displayed."

[0805] As described above, the present invention provides an integrated system that includes everything from analyzing image data to recognizing user emotions and reflecting feedback, thereby improving work efficiency and accuracy.

[0806] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0807] Step 1:

[0808] Image capture and upload

[0809] Input: User instruction to take an image

[0810] How it works: A user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other object. The user launches the application, selects the camera function, frames the object, and taps the capture button.

[0811] Output: Image data stored in the device

[0812] Step 2:

[0813] Select and upload an image

[0814] Input: Image data stored on the device

[0815] How it works: The user selects an image they have taken in the application and taps the "Upload Image" button. This action causes the device to temporarily store the selected image data and prepare it for transmission to the server.

[0816] Output: Image data ready to send to the server

[0817] Step 3:

[0818] Image data preprocessing

[0819] Input: Selected image data

[0820] What it does: The device preprocesses the image data, using Python's OpenCV to resize the image (e.g., change the resolution to 1024x768), denoise it (apply a Gaussian filter), and perform color space conversion (from RGB to grayscale).

[0821] Output: Preprocessed image data

[0822] Step 4:

[0823] Sending data to the server

[0824] Input: Preprocessed image data

[0825] Operation: The device sends the preprocessed image data to the server as an HTTP POST request, which includes a unique identifier (e.g., a UUID).

[0826] Output: Image data received by the server

[0827] Step 5:

[0828] Re-preprocessing of image data

[0829] Input: Image data received by the server

[0830] Operation: The server performs preprocessing on the received image data again, specifically edge detection (using the Canny algorithm) and image normalization (adjusting values ​​to the 0-1 range).

[0831] Output: Re-preprocessed image data

[0832] Step 6:

[0833] Image analysis and feature extraction

[0834] Input: Re-preprocessed image data

[0835] How it works: The server inputs image data into a deep learning model using TensorFlow and extracts features, such as shapes and patterns from the image using a convolutional neural network (CNN).

[0836] Output: Features extracted from the image

[0837] Step 7:

[0838] Identifying the product name

[0839] Input: extracted features

[0840] How it works: The server searches the database based on the extracted features to identify the best-matching product name. Specifically, it compares the features with known database entries and identifies the entries with the highest similarity.

[0841] Output: Identified product name

[0842] Step 8:

[0843] Obtaining product information

[0844] Input: Identified product name

[0845] Operation: The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.). The database search uses SQL.

[0846] Output: Retrieved product information

[0847] Step 9:

[0848] Encoding and sending the results

[0849] Input: Retrieved product information

[0850] Operation: The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response.

[0851] Output: Product information sent to the terminal

[0852] Step 10:

[0853] Displaying the results

[0854] Input: Product information sent from the server

[0855] Operation: The device decodes the received JSON data and displays it in the user interface. The user can view product information on the display. For example, "This antenna is model A123. Specifications are as follows: ..."

[0856] Output: Product information displayed in the user interface

[0857] Step 11:

[0858] Emotion recognition and feedback collection

[0859] Input: User's facial expression data and voice data

[0860] How it works: The emotion engine uses the device's camera and microphone to collect the user's facial expressions and voice. It uses Python's OpenCV and TensorFlow to analyze the facial expressions and voice in real time and recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives the emotion data and adjusts the system's display and operations based on the emotion.

[0861] Output: Recognized user emotion data and the system's display content adjusted accordingly

[0862] Step 12:

[0863] Analyzing feedback and improving the system

[0864] Input: User-provided feedback and emotional data

[0865] How it works: The user taps the "Feedback" button to report problems with the displayed information and submit suggestions for correction. The server uses the emotion data to prioritize the feedback and use it to retrain the system and update the database.

[0866] Output: Improved system and updated database

[0867] Through the above processing steps, the system efficiently and accurately analyzes image data, recognizes user emotions, and collects and reflects feedback.

[0868] (Application example 2)

[0869] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0870] In recent years, quality control on factory production lines has become increasingly important, but traditional quality inspections often rely on human labor, resulting in many issues in terms of efficiency and accuracy. Furthermore, there is also the problem that feedback based on the emotions and satisfaction of inspectors is not sufficiently provided, making it difficult to improve the work environment. In response to these issues, the challenge is to provide a more efficient and highly accurate quality inspection system by utilizing image data analysis and emotion recognition.

[0871] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name from a database, means for displaying the acquired product name and related information, and means for recognizing the user's emotions and adjusting the behavior of the system. This makes it possible to improve the efficiency and accuracy of quality inspections on production lines and provide feedback based on the emotions of workers.

[0872] "Image data" refers to visual information captured using a camera or other imaging device.

[0873] "Processing means" refers to a method or device that performs preprocessing such as resizing, noise removal, and color space conversion on the acquired image data to make it suitable for analysis.

[0874] "Features" are attributes such as specific patterns, shapes, colors, and textures extracted from image data, and are information that enables product identification and classification.

[0875] "Product name" refers to the product's unique name or model number that is searched for in the database based on the identified features.

[0876] A "database" is an information collection system that stores product names and other related information based on identified features and allows them to be searched and retrieved.

[0877] A "user interface" is a screen or operation panel that allows the user to exchange information with the system, and is capable of displaying and inputting information.

[0878] The "means for recognizing emotions" refers to a method or device for analyzing emotions from a user's facial expressions and voice and identifying their emotional state.

[0879] The "means for adjusting the system's behavior" refers to a method or device for dynamically changing the system's operation or display content based on the results of emotion recognition.

[0880] The following specific steps must be taken to implement this invention. The present invention provides a system for improving the efficiency of quality control on factory production lines, combining image data analysis with user emotion recognition. This system is primarily composed of a server, a terminal, a user, and an emotion engine.

[0881] First, the user takes a picture of the product using the device's camera. After taking the picture, the user selects the image in the application and taps the "Upload Image" button. This selects the image data in the device. For example, the user takes a picture of a product on a production line and taps the "Upload Image" button.

[0882] Next, the device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of analysis on the server. The preprocessed image data is then sent to the server as an HTTP POST request.

[0883] The server receives image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to make it suitable for analysis. The preprocessed image data is input into an image recognition model. The image recognition model uses a deep learning algorithm with TensorFlow to extract features from the image and identify similar product names based on the extracted features. As a specific example, the server analyzes an uploaded product image and identifies that the image is of product "model number 123."

[0884] Next, the server searches a database based on the identified product name and retrieves the corresponding product information (product name, specifications, installation method, etc.). The retrieved product information provides all the information the user needs. The server encodes the retrieved product information in JSON format and sends it to the terminal as an HTTP response. The terminal receives this response, decodes it, and displays it on the user interface. The user can check the displayed product information. As a specific example, the server encodes the retrieved product information for "model number 123" in JSON format and sends it back to the terminal. The terminal receives this and displays, "This product is model number 123. Its specifications are as follows: ..."

[0885] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions. Specifically, it can change the content displayed in the user interface or provide additional help information. For example, if the user makes a face expressing dissatisfaction with the results, the emotion engine could recognize this and display more detailed information or additional support options.

[0886] For example, the prompt for taking a picture is:

[0887] "Take a photo of the defect in the product and upload the image. We'll find out what the defect is."

[0888] In this way, the system of the present invention can improve the efficiency and accuracy of quality control on factory production lines through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[0889] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0890] Step 1:

[0891] A user takes a picture of a product using the device's camera. The input is a physical image of the product, and the output is digital image data stored on the device. Specifically, the user opens an application on the device, uses the camera function to take a picture of the product, and saves the image in the application.

[0892] Step 2:

[0893] The user selects an image in the application and taps the "Upload Image" button. The input is the image data saved in step 1, and the output is the unprocessed image data ready to be sent from the device application to the server. Specifically, the user selects an image file in the application and taps the "Upload Image" button.

[0894] Step 3:

[0895] The device receives the selected image data and performs preprocessing. The input is the image data selected by the user, and the output is the preprocessed image data. Specifically, the device performs image resizing, noise removal, color space conversion, etc., to prepare the image for analysis. OpenCV's resize function is used for resizing, and blur or GaussianBlur functions are used for noise removal.

[0896] Step 4:

[0897] The preprocessed image data is sent to the server as an HTTP POST request. The input is the preprocessed image data, and the output is the data to be sent to the server. In specific operations, the terminal generates an HTTP POST request and sends the request including the preprocessed image data to the server.

[0898] Step 5:

[0899] The server receives the image data sent from the device and preprocesses it again. The input is the preprocessed image data sent from the device, and the output is image data suitable for analysis. Specifically, the server performs edge detection and image normalization, preparing it for input into the TensorFlow deep learning model.

[0900] Step 6:

[0901] The server uses a deep learning model to extract features from images and identify product names based on the identified features. The input is preprocessed image data, and the output is the identified product name. Specifically, the server uses TensorFlow to analyze images, detect specific features, and identify product names based on them.

[0902] Step 7:

[0903] The server searches the database based on the identified product name and retrieves the corresponding product information. The input is the identified product name and the output is the retrieved product information. Specifically, the server executes a database query to retrieve information such as the name, specifications, and installation method of the corresponding product.

[0904] Step 8:

[0905] The product information acquired by the server is encoded in JSON format and sent to the terminal as an HTTP response. The input is the acquired product information, and the output is JSON format data. Specifically, the server converts the acquired product information into JSON and sends it to the terminal as an HTTP response.

[0906] Step 9:

[0907] The terminal receives the response from the server, decodes it, and displays it on the user interface. The input is the JSON data returned from the server, and the output is the product information displayed on the user interface. Specifically, the terminal parses the JSON data and displays the product information on the screen.

[0908] Step 10:

[0909] The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions. The input is the user's facial expressions and voice data, and the output is the recognized emotion data. Specifically, the device or server analyzes the user's emotions using Haar Cascade or other emotion recognition algorithms.

[0910] Step 11:

[0911] The server adjusts the system behavior based on the emotion recognition results. The input is the recognized emotion data, and the output is the adjusted system behavior or additional display information. Specific actions include changing the display content of the user interface or providing additional help information.

[0912] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0913] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0914] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0915] [Third embodiment]

[0916] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0917] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0918] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0919] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0920] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0921] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0922] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0923] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0924] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0925] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0926] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0927] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0928] In order to implement this invention, the following specific steps must be taken.

[0929] System configuration

[0930] This system is mainly composed of three components: a server, a terminal, and a user. The server provides a back-end environment for advanced analysis, while the terminal functions as an interface with the user. The user operates the terminal to collect image data and identifies components through the system.

[0931] Program processing flow

[0932] The specific flow of program processing in this system will be explained below.

[0933] 1. Taking and uploading images

[0934] The user uses the device's camera to take photos of components such as radios, antennas, and concrete poles. The captured images are selected within the device's application and sent to the server using the "Image Upload" function. The device then preprocesses the image data (resizing, noise removal, etc.) before preparing it for transfer to the server.

[0935] As a specific example, a user can take a picture of a base station antenna and tap the "upload image" button, which sends the image to the server.

[0936] 2. Receiving and preprocessing image data

[0937] The server receives the image data sent from the device. This image data undergoes further preprocessing (e.g., color space conversion, edge detection) to prepare it for more accurate analysis. Once preprocessed, the image data is used in the next step.

[0938] 3. Analysis using image recognition models

[0939] The preprocessed image data is input into the server's image recognition model, which uses machine learning algorithms such as deep learning to extract features from the image. Based on the extracted features, the server identifies the most similar product name. This identified product name is used in the next step.

[0940] As a specific example, the server analyzes an uploaded image of an antenna and identifies that the image is an antenna of "Model A123."

[0941] 4. Obtaining product name and related information

[0942] The server uses the identified product name to retrieve product information (product name, specifications, installation method, etc.) from the database. This information is very important for users to quickly obtain the information they need.

[0943] 5. Returning and displaying results

[0944] The server encodes the acquired product information in JSON format and sends it back to the device. The device receives this response, decodes it, and displays it in the user interface. The user can check the displayed information and take further action as necessary.

[0945] As a concrete example, the server encodes the antenna information for "Model A123" in JSON format and sends it back to the device. The device receives this and displays "This antenna is model A123. The specifications are as follows: ..."

[0946] User Feedback

[0947] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server, which receives and analyzes this feedback to help improve the system.

[0948] Specifically, users tap the "Feedback" button, write down the problems and suggestions for fixes, and submit them. The server analyzes this information and uses it to retrain the system and update the database.

[0949] In this way, the present invention improves work efficiency by quickly and accurately identifying base station components using image data. Even inexperienced newcomers and transferred employees can immediately contribute, and the overall work flow proceeds smoothly.

[0950] The above is a specific embodiment for carrying out the invention.

[0951] The processing flow will be explained below.

[0952] Step 1:

[0953] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data on the device.

[0954] Step 2:

[0955] The device receives the selected image data and performs preprocessing, such as resizing the image, removing noise, and converting the color space, to prepare it for analysis. This preprocessing improves the accuracy of the analysis on the server.

[0956] Step 3:

[0957] The device sends the preprocessed image data to the server as an HTTP POST request to the server's API endpoint, where the image data is encoded and transferred.

[0958] Step 4:

[0959] The server receives the image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to prepare the image data in a state suitable for analysis.

[0960] Step 5:

[0961] The server inputs the preprocessed image data into an image recognition model, which uses a deep learning algorithm to extract features from the image and identify similar product names based on the extracted features.

[0962] Step 6:

[0963] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.), thereby collecting all the information the user needs.

[0964] Step 7:

[0965] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response, which prepares the terminal to receive data from the server.

[0966] Step 8:

[0967] The terminal receives and decodes the JSON data returned by the server. The decoded data is displayed in the user interface within the application, allowing the user to check the displayed product information.

[0968] Step 9:

[0969] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Specifically, the user taps the "Feedback" button, enters the problem and a correction suggestion, and submits it.

[0970] Step 10:

[0971] The server receives and analyzes the feedback data sent by users. Based on the analysis results, it retrains the system and image recognition model, updates the database, and aims to improve the accuracy of the entire system.

[0972] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes from capturing image data to analyzing, displaying the results, and collecting feedback.

[0973] Example 1

[0974] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0975] Conventional product identification systems using image data have the problem of being prone to misidentification due to low analysis accuracy on the server side, making it difficult to obtain accurate product information. Furthermore, the lack of a user feedback function makes it difficult to improve the system. Therefore, there is a need for the development of a system that can process image data quickly and accurately and provide product information to users.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0977] In this invention, the server includes means for preprocessing image data, means for transmitting the preprocessed image data to the server, means for further processing the image data received by the server, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name and related information from a database, means for encoding and transmitting the acquired product name and related information, means for decoding and displaying the transmitted information, and means for receiving and analyzing feedback from a user. This makes it possible to quickly and accurately identify products using image data and improve business efficiency.

[0978] "Image data" is data that represents visual information in a digital format.

[0979] "Preprocessing" refers to processes such as resizing, noise removal, and color space conversion that are carried out to make image data easier to analyze.

[0980] A "server" is a computer system that processes and stores data and communicates with clients.

[0981] A "terminal" is a device such as a computer or smartphone that is operated by a user.

[0982] A "feature" is information that is extracted from image data and represents a specific property or pattern.

[0983] "Product name" is a name used to identify a specific product.

[0984] A "database" is a system that allows for efficient management and retrieval of organized collections of data.

[0985] "Encoding" is the process of converting data into a certain format.

[0986] "Decoding" is the process of restoring encoded data to its original form.

[0987] "Feedback" is the act of sending a user's evaluation or opinion about a system.

[0988] To implement this invention, the following specific steps must be taken. First, the system is composed of three entities: a server, a terminal, and a user. The server provides a back-end environment for performing advanced analysis, and the terminal functions as an interface with the user. The user operates the terminal to collect image data and identifies components through the system.

[0989] System configuration

[0990] Specific examples of the hardware and software used in this system are shown below.

[0991] Server: AWS EC2 instance

[0992] Device: Smartphone or tablet

[0993] Preprocessing tool: OpenCV

[0994] Image recognition model: TensorFlow

[0995] Database: MySQL

[0996] JSON encoding: Jackson library (Java)

[0997] UI display: React Native application

[0998] Program processing

[0999] First, the user uses the device's camera app to take a photo of a radio, antenna, concrete pole, or other component. The user selects the image and taps the "Upload Image" button to send it to the server. The device then performs preprocessing such as resizing and noise removal on the image data before preparing it for transfer.

[1000] The server receives the image data sent from the device and performs further advanced preprocessing (e.g., color space conversion, edge detection). This processing improves the accuracy of image analysis. After preprocessing, the image data is input into the server's image recognition model.

[1001] The server's image recognition model uses a deep learning algorithm, specifically a Convolutional Neural Network (CNN), to extract features from images. Based on the extracted features, the server identifies the most similar product name.

[1002] Using the identified product name, the server retrieves related product information (such as name, specifications, and installation method) from the database. This information is encoded in JSON format and sent back to the terminal. The terminal decodes the received JSON data and displays it on the user interface. The user can check the displayed information and take further action as needed.

[1003] Furthermore, if a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request, which the server receives and analyzes to help improve the system.

[1004] Specific examples

[1005] As a specific example of how it works, a user takes a photo of a base station antenna and taps the "Upload Image" button, which sends the image to the server. The server analyzes the uploaded antenna image and identifies it as a "Model A123" antenna. It then encodes the "Model A123" antenna information it retrieves from the database in JSON format and returns it to the device. The device that receives this displays the message, "This antenna is Model A123. The specifications are as follows: ..."

[1006] Prompt Sentence Examples

[1007] The user taps the "Feedback" button and sends a comment saying, "The displayed information is incorrect. The actual model is B456." The server receives this, analyzes the feedback, and uses it for retraining and updating the database.

[1008] In this way, the present invention uses image data to quickly and accurately identify base station components, improving work efficiency and enabling even inexperienced new employees or transferred employees to immediately contribute to the company.

[1009] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1010] Step 1:

[1011] The user launches the camera app on their device and takes a photo of a radio, antenna, concrete pole, or other component. The image is then saved in the app.

[1012] Input: Physical image of the part

[1013] Output: Digital image data stored on the device

[1014] Specific behavior:

[1015] The user opens the camera app on their smartphone and takes a picture of the antenna. The image is then saved on the device.

[1016] Step 2:

[1017] The user selects the image they have taken in the application and taps the "Upload Image" button. The device then performs preprocessing on the selected image, such as resizing and noise reduction.

[1018] Input: Digital image data stored on the device

[1019] Output: Pre-processed digital image data

[1020] Specific behavior:

[1021] The user selects an image in the app and taps the "Upload Image" button. The device resizes and denoises the image.

[1022] Step 3:

[1023] The terminal transmits the preprocessed image data to the server.

[1024] Input: Preprocessed digital image data

[1025] Output: Preprocessed image data sent to the server

[1026] Specific behavior:

[1027] The terminal transmits the preprocessed image data to the server.

[1028] Step 4:

[1029] The server receives the image data sent from the terminal, and then performs further preprocessing on this image data (color space conversion, edge detection).

[1030] Input: Preprocessed image data

[1031] Output: Image data processed with edge detection etc.

[1032] Specific behavior:

[1033] The server receives the image sent from the device, converts the color space from RGB to grayscale, and applies an edge detection algorithm to highlight the features of the components.

[1034] Step 5:

[1035] The server inputs the preprocessed image data into the image recognition model, which uses a deep learning algorithm (CNN) to extract features from the image.

[1036] Input: Preprocessed image data

[1037] Output: Features extracted from the recognition model

[1038] Specific behavior:

[1039] The server inputs the image data into the machine learning model, and the CNN extracts image features through filters.

[1040] Step 6:

[1041] The server identifies the most similar product name based on the extracted features.

[1042] Input: Extracted features

[1043] Output: Identified product name

[1044] Specific behavior:

[1045] Based on the extracted features, the machine learning model identifies the uploaded antenna as "Model A123."

[1046] Step 7:

[1047] The server queries the database using the identified product name to retrieve relevant product information (such as name, specifications, installation instructions, etc.).

[1048] Input: Identified product name

[1049] Output: Product information retrieved from the database

[1050] Specific behavior:

[1051] The server requests information from the MySQL database using the product name "Model A123", and the database returns the relevant data to the server.

[1052] Step 8:

[1053] The server encodes the acquired product information in JSON format and sends it to the terminal.

[1054] Input: Product information retrieved from the database

[1055] Output: JSON encoded data

[1056] Specific behavior:

[1057] The server converts the product information into JSON format and sends it to the terminal.

[1058] Step 9:

[1059] The terminal decodes the received JSON data and displays it in the user interface.

[1060] Input: JSON format data

[1061] Output: Product information displayed in the user interface

[1062] Specific behavior:

[1063] The device analyzes the received data and displays on the user interface, "This antenna is model A123. The specifications are as follows: ..."

[1064] Step 10:

[1065] If the user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server.

[1066] Input: User feedback

[1067] Output: Feedback data sent to the server

[1068] Specific behavior:

[1069] The user taps the "Feedback" button and submits the message stating, "The displayed information is incorrect. The actual model is B456."

[1070] In this way, the system uses image data to accurately identify products and provide information quickly.

[1071] (Application example 1)

[1072] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1073] In modern manufacturing, parts management and maintenance are important, but it is difficult to centrally manage and quickly identify a huge number of parts. To solve this problem, it is necessary to accurately and efficiently identify parts and obtain maintenance information while minimizing manual work.

[1074] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1075] In this invention, the server includes means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying an item name based on the extracted features, means for acquiring the identified item name from a database, means for displaying the acquired item name and related information, means for acquiring and transmitting component data using an image acquisition device mounted on a factory machine, and means for analyzing the transmitted data and automatically returning the item name and related information, thereby enabling accurate and efficient acquisition of component identification and maintenance information.

[1076] "Image data" refers to visual information captured using a camera or other imaging device.

[1077] "Processing means" refers to algorithms or programs for analyzing or converting image data.

[1078] A "feature amount" is an important attribute or index for identifying a component extracted from image data.

[1079] "Item name" refers to the official name or identification label of the identified component.

[1080] A "database" is a storage system that stores product names and related information and allows them to be searched and retrieved.

[1081] "Related information" is detailed data such as specifications, installation methods, and maintenance information related to components.

[1082] The "means for displaying" refers to a display or interface that allows the user to visually recognize the acquired product name and related information.

[1083] "Factory machinery" refers to industrial equipment and robots used on production lines and for maintenance work.

[1084] "Image capture devices" refer to cameras and sensors mounted on factory machinery.

[1085] "Transmission means" refers to a communication function for transmitting image data and analysis results to a server.

[1086] The "analysis means" refers to an algorithm or program used on the server side to extract item names and features from image data.

[1087] The "automatic return method" is a communication function that sends the item name and related information generated based on the analysis results from the server to the user.

[1088] MODE FOR CARRYING OUT THE INVENTION

[1089] To implement the present invention, the following system configuration and processing are required.

[1090] System Configuration

[1091] The system is primarily composed of three components: a server, a terminal, and a user. The server provides a back-end environment for advanced analysis, while the terminal functions as an interface with the user. The user operates the terminal to collect image data of components and obtains the item names and related information identified through the system.

[1092] Program processing flow

[1093] 1. Acquiring and uploading images

[1094] A photo of the component is taken using the image capture device of the factory machine installed on the terminal. The captured image is sent to the server via the terminal's application. The image is then pre-processed, with resizing and noise removal.

[1095] 2. Receiving and analyzing image data

[1096] The server receives the image data sent from the device and performs a detailed analysis using an image recognition model and deep learning algorithm, which extracts features from the image and identifies the item name.

[1097] 3. Obtaining and displaying product names and related information

[1098] Based on the identified item name, the server retrieves relevant information from a database, which is then displayed in the user interface.

[1099] Hardware and Software

[1100] The system uses the following hardware and software:

[1101] Hardware: Camera (image capture device) mounted on factory machinery

[1102] software:

[1103] OpenCV: A library for image capture and preprocessing

[1104] requests: A library for sending HTTP requests

[1105] JSON: The format in which data received from the server is decoded and displayed.

[1106] Deep learning frameworks (e.g., TensorFlow, PyTorch): for feature extraction and running image recognition models

[1107] Specific examples

[1108] As a concrete example, consider a factory robot identifying a new concrete pole. The robot's camera takes a picture of the pole and uploads the image to a server through an application. The server analyzes the image and sends the identified item name and related information back to the robot, allowing it to execute the appropriate maintenance process.

[1109] Prompt Sentence Examples

[1110] "Please create a program for an application that analyzes images of concrete pillars taken by a factory robot with a camera and obtains the item name and specifications."

[1111] In this way, the entire system works together to execute a series of processes from acquiring image data to analyzing it and displaying the information, thereby streamlining component management and maintenance work within the factory.

[1112] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1113] Program processing steps

[1114] Step 1:

[1115] The user takes a photo of the part using an image capture device of the factory machine mounted on the terminal.

[1116] Input: Image of a part taken by a factory machine camera

[1117] Specific operation: The user operates an application on the device and captures an image with the camera.

[1118] Step 2:

[1119] The captured images are pre-processed on the device, including resizing and noise removal.

[1120] Input: Raw image data captured

[1121] Specific operation: The device application uses OpenCV to resize the image and remove noise.

[1122] Step 3:

[1123] The pre-processed image data is uploaded from the terminal to a server.

[1124] Input: Preprocessed image data

[1125] Specific operation: The terminal uses the requests library to send image data to the server as a POST request.

[1126] Step 4:

[1127] The server again preprocesses the received image data and prepares it for analysis.

[1128] Input: Preprocessed image data received from the device

[1129] Specific operation: The server uses OpenCV to perform additional preprocessing such as color space conversion and edge detection.

[1130] Step 5:

[1131] The server analyzes the preprocessed images using a deep learning framework (e.g., TensorFlow, PyTorch) and extracts features.

[1132] Input: Image data preprocessed for analysis

[1133] Specific operation: The server uses the generative AI model to extract features from image data.

[1134] Step 6:

[1135] Based on the extracted features, the server identifies the item name.

[1136] Input: extracted features

[1137] Specific operation: The server compares the features with the database and identifies the most similar item name.

[1138] Step 7:

[1139] The server retrieves relevant information from a database based on the identified item name.

[1140] Input: Identified item name

[1141] Specific behavior: The server executes a database (e.g., SQL database) query to retrieve the item name and related information.

[1142] Step 8:

[1143] The server encodes the item name and related information in JSON format and returns it to the terminal.

[1144] Input: Item name and related information obtained

[1145] Specific operation: The server encodes the relevant information in JSON format and sends it to the terminal as an HTTP response.

[1146] Step 9:

[1147] The terminal decodes the JSON data received from the server and displays it in the user interface.

[1148] Input: JSON format data received from the server

[1149] Specific behavior: The device decodes the JSON and updates the application screen to show the item name and related information to the user.

[1150] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1151] To implement this invention, the following specific steps must be taken. The present invention provides a system that improves the efficiency of base station construction work by combining image data analysis and user emotion recognition. This system is mainly composed of a server, terminals, users, and an emotion engine.

[1152] System configuration

[1153] The system consists of the following elements:

[1154] 1. Server: Responsible for analyzing image data and processing emotion data.

[1155] 2. Terminal: Captures and preprocesses image data and provides an interface with the user.

[1156] 3. User: An operator who operates the system and provides image data.

[1157] 4. Emotion engine: An engine that recognizes the user's emotions and adjusts the system's behavior based on those emotions.

[1158] Program processing flow

[1159] The specific flow of program processing in this system will be explained below.

[1160] 1. Taking and uploading images

[1161] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data in the device.

[1162] As a specific example, a user takes a picture of a base station antenna and taps the "upload image" button.

[1163] 2. Image data preprocessing and transmission

[1164] The device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of analysis on the server. The preprocessed image data is then sent to the server as an HTTP POST request.

[1165] 3. Receiving and analyzing image data

[1166] The server receives the image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to make it suitable for analysis. The preprocessed image data is then input into an image recognition model. The image recognition model uses a deep learning algorithm to extract features from the image and identify similar product names based on the extracted features.

[1167] As a specific example, the server analyzes an uploaded image of an antenna and identifies that the image is an antenna of "Model A123."

[1168] 4. Obtaining product name and related information

[1169] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.) The retrieved product information provides all the information the user needs.

[1170] 5. Returning and displaying results

[1171] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response. The terminal receives this response, decodes it, and displays it on the user interface. The user can then check the displayed product information.

[1172] As a concrete example, the server encodes the antenna information for "Model A123" in JSON format and sends it back to the device. The device receives this and displays "This antenna is model A123. The specifications are as follows: ..."

[1173] 6. Emotion Recognition and Feedback

[1174] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions. Specifically, it changes the content displayed in the user interface or provides additional help information.

[1175] For example, if a user makes a face expressing dissatisfaction with a result, the emotion engine could recognize this and display more detailed information or additional support options.

[1176] 7. Feedback Collection and Analysis

[1177] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Based on the emotional data acquired by the emotion engine, the priority of the feedback content is determined and work is done to improve the system as necessary.

[1178] For example, a user can tap a "Feedback" button, write down the problem and suggest fixes, and the server will analyze the feedback to help retrain the system and update the database.

[1179] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[1180] The processing flow will be explained below.

[1181] Step 1:

[1182] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data on the device.

[1183] Step 2:

[1184] The device receives the selected image data and performs preprocessing, such as resizing the image, removing noise, and converting the color space, to prepare it for analysis. This preprocessing improves the accuracy of the analysis on the server.

[1185] Step 3:

[1186] The device sends the preprocessed image data to the server as an HTTP POST request to the server's API endpoint, where the image data is encoded and transferred.

[1187] Step 4:

[1188] The server receives the image data sent from the device, and then preprocesses the image again (for example, by edge detection or image normalization) to prepare it for analysis.

[1189] Step 5:

[1190] The server inputs the preprocessed image data into an image recognition model, which uses a deep learning algorithm to extract features from the image and identify the most similar product name based on the extracted features.

[1191] Step 6:

[1192] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.) This information provides all the information the user needs.

[1193] Step 7:

[1194] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response, which prepares the terminal to receive data from the server.

[1195] Step 8:

[1196] The terminal receives and decodes the JSON data returned by the server. The decoded data is displayed in the application's user interface, allowing the user to check the displayed product information.

[1197] Step 9:

[1198] The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions.

[1199] Step 10:

[1200] The server adjusts the user interface display based on the recognized emotion, such as by displaying a screen offering additional help information or detailed explanations if the user expresses dissatisfaction.

[1201] Step 11:

[1202] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Based on the emotional data acquired by the emotion engine, the priority of the feedback content is determined and work is done to improve the system as necessary.

[1203] Step 12:

[1204] The server receives and analyzes the feedback data sent by users. Based on the analysis results, it retrains the system and image recognition model, updates the database, and aims to improve the accuracy of the entire system.

[1205] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[1206] Example 2

[1207] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1208] In conventional systems, image data analysis and user emotion recognition are often processed separately, resulting in insufficient information integration, reduced analysis accuracy, and difficulty in effectively incorporating user feedback. Furthermore, while base station construction work requires the rapid identification of various components and the provision of related information, there has been a lack of systems that can address this need. The present invention aims to solve these problems by providing an efficient system that integrates image data analysis, user emotion recognition, and feedback collection and reflection.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1210] In this invention, the server includes means for preprocessing the image data again and extracting features, means for identifying a product name based on the extracted features, and means for the server to search a database based on the identified product name and obtain corresponding product information. This allows for efficient integration of multiple processing steps, providing immediate feedback to the user, and enabling system adjustment based on the user's emotions.

[1211] "Means for acquiring image data" refers to a device or method for collecting visual information as digital data using an input device such as a camera or scanner.

[1212] "Preprocessing means" refers to a method or device that performs operations such as resizing, noise removal, and color space conversion on acquired image data before analysis, thereby preparing the data in a form suitable for analysis.

[1213] The "means for transmitting to a server" refers to a device or method for transferring image data processed by a terminal to a remote server via a network.

[1214] A "means for extracting features" is a method or device that identifies important patterns and attributes such as shape, color, and texture from image data and extracts them as quantities.

[1215] The "means for identifying product names" refers to a method or device that compares the extracted features with a known database and identifies the product or object that best matches.

[1216] "Means for searching a database and obtaining relevant product information" refers to a method or device for searching a database using a specified product name or other key and extracting related information (product name, specifications, installation method, etc.).

[1217] The "means for transmitting the acquired product name and related information from the server to the terminal" refers to a device or method for transferring the information searched and acquired by the server to the terminal via the network.

[1218] "Means for displaying on a user interface" refers to a device or method for visually displaying acquired information on a terminal display or screen.

[1219] "Means for analyzing a user's emotions and adjusting the system's behavior based on those emotions" refers to a method or device that analyzes a user's facial expressions and voice to identify their emotional state, and flexibly changes the system's processing, such as changing the user interface or providing additional information, depending on those emotions.

[1220] To implement this invention, the following specific steps must be taken. This system combines image data analysis and user emotion recognition to streamline specific tasks. Specifically, it includes a series of tasks from acquiring image data, analyzing it to identifying product names, providing related information based on that information, and collecting feedback based on the user's emotions. This system consists of the following main components:

[1221] System Components

[1222] 1. Server:

[1223] The server acts as the main processing unit, preprocessing image data, extracting features, identifying product names, acquiring related information, and analyzing user emotion data. The server achieves this processing by receiving and processing HTTP requests using Python's Flask and running deep learning models using TensorFlow.

[1224] 2. Terminal:

[1225] The device is responsible for acquiring and preprocessing image data, sending the data to the server, displaying the results obtained from the server, and collecting user emotion data. The device is equipped with a camera and microphone, and uses Python's OpenCV to perform preprocessing such as image resizing, noise reduction, and color space conversion.

[1226] 3. User:

[1227] The user operates the device to take and upload image data, check the displayed results, and provide necessary feedback. The user's emotional data is analyzed by the emotion engine, and the system's behavior is adjusted accordingly.

[1228] 4. Emotion Engine:

[1229] The emotion engine recognizes emotions from the user's facial expressions and voice and captures them as data. For example, a model using Python's OpenCV or TensorFlow analyzes facial expression data entered by the user through the device's camera and recognizes the emotion (satisfaction, dissatisfaction, surprise, etc.). This data is used to adjust the system's display content and operations according to the user's emotions.

[1230] Specific examples of operation

[1231] The following example shows the specific operation of the system.

[1232] Example 1: Taking and uploading image data

[1233] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component, selects the image in the application, and taps the "Upload Image" button. For example, the user takes a photo of a base station antenna and taps the "Upload Image" button.

[1234] Example 2: Preprocessing and sending image data

[1235] The device resizes the captured image data, removes noise, and converts the color space, and then sends the preprocessed image data to the server as an HTTP POST request.

[1236] Example 3: Receiving and analyzing image data

[1237] The server receives the image data sent from the device, performs preprocessing again (edge ​​detection and image normalization), and then uses a deep learning model to extract features from the image and identify the product name based on those features. For example, the server may identify an image of an antenna as "Model A123."

[1238] Example 4: Retrieving product names and related information

[1239] The server searches the database based on the identified product name and obtains the relevant product information (product name, specifications, installation method, etc.).

[1240] Example 5: Returning and displaying results

[1241] The server encodes the acquired product information in JSON format and sends it to the device as an HTTP response. The device receives this, decodes it, and displays it in the user interface. For example, it might say, "This antenna is model A123. The specifications are as follows: ..."

[1242] Example 6: Emotion Recognition and Feedback

[1243] The emotion engine analyzes the user's facial expressions and voice to recognize emotions, which the server receives and adjusts the system's behavior. For example, if the user makes a dissatisfied face, the emotion engine will recognize this and display additional information or support options.

[1244] Example 7: Feedback collection and analysis

[1245] If the displayed information is incorrect, the user can use the feedback feature to send a correction request to the server, which analyzes the feedback and uses it to retrain the system and update the database.

[1246] Prompt Sentence Examples

[1247] "This antenna is model A123. Its specifications are as follows: ..."

[1248] Please provide feedback:

[1249] "Analysis results from the emotion engine: The user's emotion was recognized as dissatisfied. Additional supporting information is displayed."

[1250] As described above, the present invention provides an integrated system that includes everything from analyzing image data to recognizing user emotions and reflecting feedback, thereby improving work efficiency and accuracy.

[1251] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1252] Step 1:

[1253] Image capture and upload

[1254] Input: User instruction to take an image

[1255] How it works: A user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other object. The user launches the application, selects the camera function, frames the object, and taps the capture button.

[1256] Output: Image data stored in the device

[1257] Step 2:

[1258] Select and upload an image

[1259] Input: Image data stored on the device

[1260] How it works: The user selects an image they have taken in the application and taps the "Upload Image" button. This action causes the device to temporarily store the selected image data and prepare it for transmission to the server.

[1261] Output: Image data ready to send to the server

[1262] Step 3:

[1263] Image data preprocessing

[1264] Input: Selected image data

[1265] What it does: The device preprocesses the image data, using Python's OpenCV to resize the image (e.g., change the resolution to 1024x768), denoise it (apply a Gaussian filter), and perform color space conversion (from RGB to grayscale).

[1266] Output: Preprocessed image data

[1267] Step 4:

[1268] Sending data to the server

[1269] Input: Preprocessed image data

[1270] Operation: The device sends the preprocessed image data to the server as an HTTP POST request, which includes a unique identifier (e.g., a UUID).

[1271] Output: Image data received by the server

[1272] Step 5:

[1273] Re-preprocessing of image data

[1274] Input: Image data received by the server

[1275] Operation: The server performs preprocessing on the received image data again, specifically edge detection (using the Canny algorithm) and image normalization (adjusting values ​​to the 0-1 range).

[1276] Output: Re-preprocessed image data

[1277] Step 6:

[1278] Image analysis and feature extraction

[1279] Input: Re-preprocessed image data

[1280] How it works: The server inputs image data into a deep learning model using TensorFlow and extracts features, such as shapes and patterns from the image using a convolutional neural network (CNN).

[1281] Output: Features extracted from the image

[1282] Step 7:

[1283] Identifying the product name

[1284] Input: extracted features

[1285] How it works: The server searches the database based on the extracted features to identify the best-matching product name. Specifically, it compares the features with known database entries and identifies the entries with the highest similarity.

[1286] Output: Identified product name

[1287] Step 8:

[1288] Obtaining product information

[1289] Input: Identified product name

[1290] Operation: The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.). The database search uses SQL.

[1291] Output: Retrieved product information

[1292] Step 9:

[1293] Encoding and sending the results

[1294] Input: Retrieved product information

[1295] Operation: The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response.

[1296] Output: Product information sent to the terminal

[1297] Step 10:

[1298] Displaying the results

[1299] Input: Product information sent from the server

[1300] Operation: The device decodes the received JSON data and displays it in the user interface. The user can view product information on the display. For example, "This antenna is model A123. Specifications are as follows: ..."

[1301] Output: Product information displayed in the user interface

[1302] Step 11:

[1303] Emotion recognition and feedback collection

[1304] Input: User's facial expression data and voice data

[1305] How it works: The emotion engine uses the device's camera and microphone to collect the user's facial expressions and voice. It uses Python's OpenCV and TensorFlow to analyze the facial expressions and voice in real time and recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives the emotion data and adjusts the system's display and operations based on the emotion.

[1306] Output: Recognized user emotion data and the system's display content adjusted accordingly

[1307] Step 12:

[1308] Analyzing feedback and improving the system

[1309] Input: User-provided feedback and emotional data

[1310] How it works: The user taps the "Feedback" button to report problems with the displayed information and submit suggestions for correction. The server uses the emotion data to prioritize the feedback and use it to retrain the system and update the database.

[1311] Output: Improved system and updated database

[1312] Through the above processing steps, the system efficiently and accurately analyzes image data, recognizes user emotions, and collects and reflects feedback.

[1313] (Application example 2)

[1314] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1315] In recent years, quality control on factory production lines has become increasingly important, but traditional quality inspections often rely on human labor, resulting in many issues in terms of efficiency and accuracy. Furthermore, there is also the problem that feedback based on the emotions and satisfaction of inspectors is not sufficiently provided, making it difficult to improve the work environment. In response to these issues, the challenge is to provide a more efficient and highly accurate quality inspection system by utilizing image data analysis and emotion recognition.

[1316] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name from a database, means for displaying the acquired product name and related information, and means for recognizing the user's emotions and adjusting the behavior of the system. This makes it possible to improve the efficiency and accuracy of quality inspections on production lines and provide feedback based on the emotions of workers.

[1317] "Image data" refers to visual information captured using a camera or other imaging device.

[1318] "Processing means" refers to a method or device that performs preprocessing such as resizing, noise removal, and color space conversion on the acquired image data to make it suitable for analysis.

[1319] "Features" are attributes such as specific patterns, shapes, colors, and textures extracted from image data, and are information that enables product identification and classification.

[1320] "Product name" refers to the product's unique name or model number that is searched for in the database based on the identified features.

[1321] A "database" is an information collection system that stores product names and other related information based on identified features and allows them to be searched and retrieved.

[1322] A "user interface" is a screen or operation panel that allows the user to exchange information with the system, and is capable of displaying and inputting information.

[1323] The "means for recognizing emotions" refers to a method or device for analyzing emotions from a user's facial expressions and voice and identifying their emotional state.

[1324] The "means for adjusting the system's behavior" refers to a method or device for dynamically changing the system's operation or display content based on the results of emotion recognition.

[1325] The following specific steps must be taken to implement this invention. The present invention provides a system for improving the efficiency of quality control on factory production lines, combining image data analysis with user emotion recognition. This system is primarily composed of a server, a terminal, a user, and an emotion engine.

[1326] First, the user takes a picture of the product using the device's camera. After taking the picture, the user selects the image in the application and taps the "Upload Image" button. This selects the image data in the device. For example, the user takes a picture of a product on a production line and taps the "Upload Image" button.

[1327] Next, the device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of analysis on the server. The preprocessed image data is then sent to the server as an HTTP POST request.

[1328] The server receives image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to make it suitable for analysis. The preprocessed image data is input into an image recognition model. The image recognition model uses a deep learning algorithm with TensorFlow to extract features from the image and identify similar product names based on the extracted features. As a specific example, the server analyzes an uploaded product image and identifies that the image is of product "model number 123."

[1329] Next, the server searches a database based on the identified product name and retrieves the corresponding product information (product name, specifications, installation method, etc.). The retrieved product information provides all the information the user needs. The server encodes the retrieved product information in JSON format and sends it to the terminal as an HTTP response. The terminal receives this response, decodes it, and displays it on the user interface. The user can check the displayed product information. As a specific example, the server encodes the retrieved product information for "model number 123" in JSON format and sends it back to the terminal. The terminal receives this and displays, "This product is model number 123. Its specifications are as follows: ..."

[1330] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions. Specifically, it can change the content displayed in the user interface or provide additional help information. For example, if the user makes a face expressing dissatisfaction with the results, the emotion engine could recognize this and display more detailed information or additional support options.

[1331] For example, the prompt for taking a picture is:

[1332] "Take a photo of the defect in the product and upload the image. We'll find out what the defect is."

[1333] In this way, the system of the present invention can improve the efficiency and accuracy of quality control on factory production lines through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[1334] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1335] Step 1:

[1336] A user takes a picture of a product using the device's camera. The input is a physical image of the product, and the output is digital image data stored on the device. Specifically, the user opens an application on the device, uses the camera function to take a picture of the product, and saves the image in the application.

[1337] Step 2:

[1338] The user selects an image in the application and taps the "Upload Image" button. The input is the image data saved in step 1, and the output is the unprocessed image data ready to be sent from the device application to the server. Specifically, the user selects an image file in the application and taps the "Upload Image" button.

[1339] Step 3:

[1340] The device receives the selected image data and performs preprocessing. The input is the image data selected by the user, and the output is the preprocessed image data. Specifically, the device performs image resizing, noise removal, color space conversion, etc., to prepare the image for analysis. OpenCV's resize function is used for resizing, and blur or GaussianBlur functions are used for noise removal.

[1341] Step 4:

[1342] The preprocessed image data is sent to the server as an HTTP POST request. The input is the preprocessed image data, and the output is the data to be sent to the server. In specific operations, the terminal generates an HTTP POST request and sends the request including the preprocessed image data to the server.

[1343] Step 5:

[1344] The server receives the image data sent from the device and preprocesses it again. The input is the preprocessed image data sent from the device, and the output is image data suitable for analysis. Specifically, the server performs edge detection and image normalization, preparing it for input into the TensorFlow deep learning model.

[1345] Step 6:

[1346] The server uses a deep learning model to extract features from images and identify product names based on the identified features. The input is preprocessed image data, and the output is the identified product name. Specifically, the server uses TensorFlow to analyze images, detect specific features, and identify product names based on them.

[1347] Step 7:

[1348] The server searches the database based on the identified product name and retrieves the corresponding product information. The input is the identified product name and the output is the retrieved product information. Specifically, the server executes a database query to retrieve information such as the name, specifications, and installation method of the corresponding product.

[1349] Step 8:

[1350] The product information acquired by the server is encoded in JSON format and sent to the terminal as an HTTP response. The input is the acquired product information, and the output is JSON format data. Specifically, the server converts the acquired product information into JSON and sends it to the terminal as an HTTP response.

[1351] Step 9:

[1352] The terminal receives the response from the server, decodes it, and displays it on the user interface. The input is the JSON data returned from the server, and the output is the product information displayed on the user interface. Specifically, the terminal parses the JSON data and displays the product information on the screen.

[1353] Step 10:

[1354] The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions. The input is the user's facial expressions and voice data, and the output is the recognized emotion data. Specifically, the device or server analyzes the user's emotions using Haar Cascade or other emotion recognition algorithms.

[1355] Step 11:

[1356] The server adjusts the system behavior based on the emotion recognition results. The input is the recognized emotion data, and the output is the adjusted system behavior or additional display information. Specific actions include changing the display content of the user interface or providing additional help information.

[1357] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1358] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1359] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1360] [Fourth embodiment]

[1361] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1362] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1363] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1364] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1365] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1366] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1367] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1368] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1369] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1370] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1371] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1372] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1373] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1374] In order to implement this invention, the following specific steps must be taken.

[1375] System configuration

[1376] This system is mainly composed of three components: a server, a terminal, and a user. The server provides a back-end environment for advanced analysis, while the terminal functions as an interface with the user. The user operates the terminal to collect image data and identifies components through the system.

[1377] Program processing flow

[1378] The specific flow of program processing in this system will be explained below.

[1379] 1. Taking and uploading images

[1380] The user uses the device's camera to take photos of components such as radios, antennas, and concrete poles. The captured images are selected within the device's application and sent to the server using the "Image Upload" function. The device then preprocesses the image data (resizing, noise removal, etc.) before preparing it for transfer to the server.

[1381] As a specific example, a user can take a picture of a base station antenna and tap the "upload image" button, which sends the image to the server.

[1382] 2. Receiving and preprocessing image data

[1383] The server receives the image data sent from the device. This image data undergoes further preprocessing (e.g., color space conversion, edge detection) to prepare it for more accurate analysis. Once preprocessed, the image data is used in the next step.

[1384] 3. Analysis using image recognition models

[1385] The preprocessed image data is input into the server's image recognition model, which uses machine learning algorithms such as deep learning to extract features from the image. Based on the extracted features, the server identifies the most similar product name. This identified product name is used in the next step.

[1386] As a specific example, the server analyzes an uploaded image of an antenna and identifies that the image is an antenna of "Model A123."

[1387] 4. Obtaining product name and related information

[1388] The server uses the identified product name to retrieve product information (product name, specifications, installation method, etc.) from the database. This information is very important for users to quickly obtain the information they need.

[1389] 5. Returning and displaying results

[1390] The server encodes the acquired product information in JSON format and sends it back to the device. The device receives this response, decodes it, and displays it in the user interface. The user can check the displayed information and take further action as necessary.

[1391] As a concrete example, the server encodes the antenna information for "Model A123" in JSON format and sends it back to the device. The device receives this and displays "This antenna is model A123. The specifications are as follows: ..."

[1392] User Feedback

[1393] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server, which receives and analyzes this feedback to help improve the system.

[1394] Specifically, users tap the "Feedback" button, write down the problems and suggestions for fixes, and submit them. The server analyzes this information and uses it to retrain the system and update the database.

[1395] In this way, the present invention improves work efficiency by quickly and accurately identifying base station components using image data. Even inexperienced newcomers and transferred employees can immediately contribute, and the overall work flow proceeds smoothly.

[1396] The above is a specific embodiment for carrying out the invention.

[1397] The processing flow will be explained below.

[1398] Step 1:

[1399] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data on the device.

[1400] Step 2:

[1401] The device receives the selected image data and performs preprocessing, such as resizing the image, removing noise, and converting the color space, to prepare it for analysis. This preprocessing improves the accuracy of the analysis on the server.

[1402] Step 3:

[1403] The device sends the preprocessed image data to the server as an HTTP POST request to the server's API endpoint, where the image data is encoded and transferred.

[1404] Step 4:

[1405] The server receives the image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to prepare the image data in a state suitable for analysis.

[1406] Step 5:

[1407] The server inputs the preprocessed image data into an image recognition model, which uses a deep learning algorithm to extract features from the image and identify similar product names based on the extracted features.

[1408] Step 6:

[1409] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.), thereby collecting all the information the user needs.

[1410] Step 7:

[1411] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response, which prepares the terminal to receive data from the server.

[1412] Step 8:

[1413] The terminal receives and decodes the JSON data returned by the server. The decoded data is displayed in the user interface within the application, allowing the user to check the displayed product information.

[1414] Step 9:

[1415] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Specifically, the user taps the "Feedback" button, enters the problem and a correction suggestion, and submits it.

[1416] Step 10:

[1417] The server receives and analyzes the feedback data sent by users. Based on the analysis results, it retrains the system and image recognition model, updates the database, and aims to improve the accuracy of the entire system.

[1418] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes from capturing image data to analyzing, displaying the results, and collecting feedback.

[1419] Example 1

[1420] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1421] Conventional product identification systems using image data have the problem of being prone to misidentification due to low analysis accuracy on the server side, making it difficult to obtain accurate product information. Furthermore, the lack of a user feedback function makes it difficult to improve the system. Therefore, there is a need for the development of a system that can process image data quickly and accurately and provide product information to users.

[1422] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1423] In this invention, the server includes means for preprocessing image data, means for transmitting the preprocessed image data to the server, means for further processing the image data received by the server, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name and related information from a database, means for encoding and transmitting the acquired product name and related information, means for decoding and displaying the transmitted information, and means for receiving and analyzing feedback from a user. This makes it possible to quickly and accurately identify products using image data and improve business efficiency.

[1424] "Image data" is data that represents visual information in a digital format.

[1425] "Preprocessing" refers to processes such as resizing, noise removal, and color space conversion that are carried out to make image data easier to analyze.

[1426] A "server" is a computer system that processes and stores data and communicates with clients.

[1427] A "terminal" is a device such as a computer or smartphone that is operated by a user.

[1428] A "feature" is information that is extracted from image data and represents a specific property or pattern.

[1429] "Product name" is a name used to identify a specific product.

[1430] A "database" is a system that allows for efficient management and retrieval of organized collections of data.

[1431] "Encoding" is the process of converting data into a certain format.

[1432] "Decoding" is the process of restoring encoded data to its original form.

[1433] "Feedback" is the act of sending a user's evaluation or opinion about a system.

[1434] To implement this invention, the following specific steps must be taken. First, the system is composed of three entities: a server, a terminal, and a user. The server provides a back-end environment for performing advanced analysis, and the terminal functions as an interface with the user. The user operates the terminal to collect image data and identifies components through the system.

[1435] System configuration

[1436] Specific examples of the hardware and software used in this system are shown below.

[1437] Server: AWS EC2 instance

[1438] Device: Smartphone or tablet

[1439] Preprocessing tool: OpenCV

[1440] Image recognition model: TensorFlow

[1441] Database: MySQL

[1442] JSON encoding: Jackson library (Java)

[1443] UI display: React Native application

[1444] Program processing

[1445] First, the user uses the device's camera app to take a photo of a radio, antenna, concrete pole, or other component. The user selects the image and taps the "Upload Image" button to send it to the server. The device then performs preprocessing such as resizing and noise removal on the image data before preparing it for transfer.

[1446] The server receives the image data sent from the device and performs further advanced preprocessing (e.g., color space conversion, edge detection). This processing improves the accuracy of image analysis. After preprocessing, the image data is input into the server's image recognition model.

[1447] The server's image recognition model uses a deep learning algorithm, specifically a Convolutional Neural Network (CNN), to extract features from images. Based on the extracted features, the server identifies the most similar product name.

[1448] Using the identified product name, the server retrieves related product information (such as name, specifications, and installation method) from the database. This information is encoded in JSON format and sent back to the terminal. The terminal decodes the received JSON data and displays it on the user interface. The user can check the displayed information and take further action as needed.

[1449] Furthermore, if a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request, which the server receives and analyzes to help improve the system.

[1450] Specific examples

[1451] As a specific example of how it works, a user takes a photo of a base station antenna and taps the "Upload Image" button, which sends the image to the server. The server analyzes the uploaded antenna image and identifies it as a "Model A123" antenna. It then encodes the "Model A123" antenna information it retrieves from the database in JSON format and returns it to the device. The device that receives this displays the message, "This antenna is Model A123. The specifications are as follows: ..."

[1452] Prompt Sentence Examples

[1453] The user taps the "Feedback" button and sends a comment saying, "The displayed information is incorrect. The actual model is B456." The server receives this, analyzes the feedback, and uses it for retraining and updating the database.

[1454] In this way, the present invention uses image data to quickly and accurately identify base station components, improving work efficiency and enabling even inexperienced new employees or transferred employees to immediately contribute to the company.

[1455] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1456] Step 1:

[1457] The user launches the camera app on their device and takes a photo of a radio, antenna, concrete pole, or other component. The image is then saved in the app.

[1458] Input: Physical image of the part

[1459] Output: Digital image data stored on the device

[1460] Specific behavior:

[1461] The user opens the camera app on their smartphone and takes a picture of the antenna. The image is then saved on the device.

[1462] Step 2:

[1463] The user selects the image they have taken in the application and taps the "Upload Image" button. The device then performs preprocessing on the selected image, such as resizing and noise reduction.

[1464] Input: Digital image data stored on the device

[1465] Output: Pre-processed digital image data

[1466] Specific behavior:

[1467] The user selects an image in the app and taps the "Upload Image" button. The device resizes and denoises the image.

[1468] Step 3:

[1469] The terminal transmits the preprocessed image data to the server.

[1470] Input: Preprocessed digital image data

[1471] Output: Preprocessed image data sent to the server

[1472] Specific behavior:

[1473] The terminal transmits the preprocessed image data to the server.

[1474] Step 4:

[1475] The server receives the image data sent from the terminal, and then performs further preprocessing on this image data (color space conversion, edge detection).

[1476] Input: Preprocessed image data

[1477] Output: Image data processed with edge detection etc.

[1478] Specific behavior:

[1479] The server receives the image sent from the device, converts the color space from RGB to grayscale, and applies an edge detection algorithm to highlight the features of the components.

[1480] Step 5:

[1481] The server inputs the preprocessed image data into the image recognition model, which uses a deep learning algorithm (CNN) to extract features from the image.

[1482] Input: Preprocessed image data

[1483] Output: Features extracted from the recognition model

[1484] Specific behavior:

[1485] The server inputs the image data into the machine learning model, and the CNN extracts image features through filters.

[1486] Step 6:

[1487] The server identifies the most similar product name based on the extracted features.

[1488] Input: Extracted features

[1489] Output: Identified product name

[1490] Specific behavior:

[1491] Based on the extracted features, the machine learning model identifies the uploaded antenna as "Model A123."

[1492] Step 7:

[1493] The server queries the database using the identified product name to retrieve relevant product information (such as name, specifications, installation instructions, etc.).

[1494] Input: Identified product name

[1495] Output: Product information retrieved from the database

[1496] Specific behavior:

[1497] The server requests information from the MySQL database using the product name "Model A123", and the database returns the relevant data to the server.

[1498] Step 8:

[1499] The server encodes the acquired product information in JSON format and sends it to the terminal.

[1500] Input: Product information retrieved from the database

[1501] Output: JSON encoded data

[1502] Specific behavior:

[1503] The server converts the product information into JSON format and sends it to the terminal.

[1504] Step 9:

[1505] The terminal decodes the received JSON data and displays it in the user interface.

[1506] Input: JSON format data

[1507] Output: Product information displayed in the user interface

[1508] Specific behavior:

[1509] The device analyzes the received data and displays on the user interface, "This antenna is model A123. The specifications are as follows: ..."

[1510] Step 10:

[1511] If the user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server.

[1512] Input: User feedback

[1513] Output: Feedback data sent to the server

[1514] Specific behavior:

[1515] The user taps the "Feedback" button and submits the message stating, "The displayed information is incorrect. The actual model is B456."

[1516] In this way, the system uses image data to accurately identify products and provide information quickly.

[1517] (Application example 1)

[1518] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1519] In modern manufacturing, parts management and maintenance are important, but it is difficult to centrally manage and quickly identify a huge number of parts. To solve this problem, it is necessary to accurately and efficiently identify parts and obtain maintenance information while minimizing manual work.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1521] In this invention, the server includes means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying an item name based on the extracted features, means for acquiring the identified item name from a database, means for displaying the acquired item name and related information, means for acquiring and transmitting component data using an image acquisition device mounted on a factory machine, and means for analyzing the transmitted data and automatically returning the item name and related information, thereby enabling accurate and efficient acquisition of component identification and maintenance information.

[1522] "Image data" refers to visual information captured using a camera or other imaging device.

[1523] "Processing means" refers to algorithms or programs for analyzing or converting image data.

[1524] A "feature amount" is an important attribute or index for identifying a component extracted from image data.

[1525] "Item name" refers to the official name or identification label of the identified component.

[1526] A "database" is a storage system that stores product names and related information and allows them to be searched and retrieved.

[1527] "Related information" is detailed data such as specifications, installation methods, and maintenance information related to components.

[1528] The "means for displaying" refers to a display or interface that allows the user to visually recognize the acquired product name and related information.

[1529] "Factory machinery" refers to industrial equipment and robots used on production lines and for maintenance work.

[1530] "Image capture devices" refer to cameras and sensors mounted on factory machinery.

[1531] "Transmission means" refers to a communication function for transmitting image data and analysis results to a server.

[1532] The "analysis means" refers to an algorithm or program used on the server side to extract item names and features from image data.

[1533] The "automatic return method" is a communication function that sends the item name and related information generated based on the analysis results from the server to the user.

[1534] MODE FOR CARRYING OUT THE INVENTION

[1535] To implement the present invention, the following system configuration and processing are required.

[1536] System Configuration

[1537] The system is primarily composed of three components: a server, a terminal, and a user. The server provides a back-end environment for advanced analysis, while the terminal functions as an interface with the user. The user operates the terminal to collect image data of components and obtains the item names and related information identified through the system.

[1538] Program processing flow

[1539] 1. Acquiring and uploading images

[1540] A photo of the component is taken using the image capture device of the factory machine installed on the terminal. The captured image is sent to the server via the terminal's application. The image is then pre-processed, with resizing and noise removal.

[1541] 2. Receiving and analyzing image data

[1542] The server receives the image data sent from the device and performs a detailed analysis using an image recognition model and deep learning algorithm, which extracts features from the image and identifies the item name.

[1543] 3. Obtaining and displaying product names and related information

[1544] Based on the identified item name, the server retrieves relevant information from a database, which is then displayed in the user interface.

[1545] Hardware and Software

[1546] The system uses the following hardware and software:

[1547] Hardware: Camera (image capture device) mounted on factory machinery

[1548] software:

[1549] OpenCV: A library for image capture and preprocessing

[1550] requests: A library for sending HTTP requests

[1551] JSON: The format in which data received from the server is decoded and displayed.

[1552] Deep learning frameworks (e.g., TensorFlow, PyTorch): for feature extraction and running image recognition models

[1553] Specific examples

[1554] As a concrete example, consider a factory robot identifying a new concrete pole. The robot's camera takes a picture of the pole and uploads the image to a server through an application. The server analyzes the image and sends the identified item name and related information back to the robot, allowing it to execute the appropriate maintenance process.

[1555] Prompt Sentence Examples

[1556] "Please create a program for an application that analyzes images of concrete pillars taken by a factory robot with a camera and obtains the item name and specifications."

[1557] In this way, the entire system works together to execute a series of processes from acquiring image data to analyzing it and displaying the information, thereby streamlining component management and maintenance work within the factory.

[1558] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1559] Program processing steps

[1560] Step 1:

[1561] The user takes a photo of the part using an image capture device of the factory machine mounted on the terminal.

[1562] Input: Image of a part taken by a factory machine camera

[1563] Specific operation: The user operates an application on the device and captures an image with the camera.

[1564] Step 2:

[1565] The captured images are pre-processed on the device, including resizing and noise removal.

[1566] Input: Raw image data captured

[1567] Specific operation: The device application uses OpenCV to resize the image and remove noise.

[1568] Step 3:

[1569] The pre-processed image data is uploaded from the terminal to a server.

[1570] Input: Preprocessed image data

[1571] Specific operation: The terminal uses the requests library to send image data to the server as a POST request.

[1572] Step 4:

[1573] The server again preprocesses the received image data and prepares it for analysis.

[1574] Input: Preprocessed image data received from the device

[1575] Specific operation: The server uses OpenCV to perform additional preprocessing such as color space conversion and edge detection.

[1576] Step 5:

[1577] The server analyzes the preprocessed images using a deep learning framework (e.g., TensorFlow, PyTorch) and extracts features.

[1578] Input: Image data preprocessed for analysis

[1579] Specific operation: The server uses the generative AI model to extract features from image data.

[1580] Step 6:

[1581] Based on the extracted features, the server identifies the item name.

[1582] Input: extracted features

[1583] Specific operation: The server compares the features with the database and identifies the most similar item name.

[1584] Step 7:

[1585] The server retrieves relevant information from a database based on the identified item name.

[1586] Input: Identified item name

[1587] Specific behavior: The server executes a database (e.g., SQL database) query to retrieve the item name and related information.

[1588] Step 8:

[1589] The server encodes the item name and related information in JSON format and returns it to the terminal.

[1590] Input: Item name and related information obtained

[1591] Specific operation: The server encodes the relevant information in JSON format and sends it to the terminal as an HTTP response.

[1592] Step 9:

[1593] The terminal decodes the JSON data received from the server and displays it in the user interface.

[1594] Input: JSON format data received from the server

[1595] Specific behavior: The device decodes the JSON and updates the application screen to show the item name and related information to the user.

[1596] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1597] To implement this invention, the following specific steps must be taken. The present invention provides a system that improves the efficiency of base station construction work by combining image data analysis and user emotion recognition. This system is mainly composed of a server, terminals, users, and an emotion engine.

[1598] System configuration

[1599] The system consists of the following elements:

[1600] 1. Server: Responsible for analyzing image data and processing emotion data.

[1601] 2. Terminal: Captures and preprocesses image data and provides an interface with the user.

[1602] 3. User: An operator who operates the system and provides image data.

[1603] 4. Emotion engine: An engine that recognizes the user's emotions and adjusts the system's behavior based on those emotions.

[1604] Program processing flow

[1605] The specific flow of program processing in this system will be explained below.

[1606] 1. Taking and uploading images

[1607] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data in the device.

[1608] As a specific example, a user takes a picture of a base station antenna and taps the "upload image" button.

[1609] 2. Image data preprocessing and transmission

[1610] The device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of analysis on the server. The preprocessed image data is then sent to the server as an HTTP POST request.

[1611] 3. Receiving and analyzing image data

[1612] The server receives the image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to make it suitable for analysis. The preprocessed image data is then input into an image recognition model. The image recognition model uses a deep learning algorithm to extract features from the image and identify similar product names based on the extracted features.

[1613] As a specific example, the server analyzes an uploaded image of an antenna and identifies that the image is an antenna of "Model A123."

[1614] 4. Obtaining product name and related information

[1615] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.) The retrieved product information provides all the information the user needs.

[1616] 5. Returning and displaying results

[1617] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response. The terminal receives this response, decodes it, and displays it on the user interface. The user can then check the displayed product information.

[1618] As a concrete example, the server encodes the antenna information for "Model A123" in JSON format and sends it back to the device. The device receives this and displays "This antenna is model A123. The specifications are as follows: ..."

[1619] 6. Emotion Recognition and Feedback

[1620] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions. Specifically, it changes the content displayed in the user interface or provides additional help information.

[1621] For example, if a user makes a face expressing dissatisfaction with a result, the emotion engine could recognize this and display more detailed information or additional support options.

[1622] 7. Feedback Collection and Analysis

[1623] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Based on the emotional data acquired by the emotion engine, the priority of the feedback content is determined and work is done to improve the system as necessary.

[1624] For example, a user can tap a "Feedback" button, write down the problem and suggest fixes, and the server will analyze the feedback to help retrain the system and update the database.

[1625] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[1626] The processing flow will be explained below.

[1627] Step 1:

[1628] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component. After taking the photo, the user selects the image in the application and taps the "Upload Image" button. This selects the image data on the device.

[1629] Step 2:

[1630] The device receives the selected image data and performs preprocessing, such as resizing the image, removing noise, and converting the color space, to prepare it for analysis. This preprocessing improves the accuracy of the analysis on the server.

[1631] Step 3:

[1632] The device sends the preprocessed image data to the server as an HTTP POST request to the server's API endpoint, where the image data is encoded and transferred.

[1633] Step 4:

[1634] The server receives the image data sent from the device, and then preprocesses the image again (for example, by edge detection or image normalization) to prepare it for analysis.

[1635] Step 5:

[1636] The server inputs the preprocessed image data into an image recognition model, which uses a deep learning algorithm to extract features from the image and identify the most similar product name based on the extracted features.

[1637] Step 6:

[1638] The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.) This information provides all the information the user needs.

[1639] Step 7:

[1640] The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response, which prepares the terminal to receive data from the server.

[1641] Step 8:

[1642] The terminal receives and decodes the JSON data returned by the server. The decoded data is displayed in the application's user interface, allowing the user to check the displayed product information.

[1643] Step 9:

[1644] The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions.

[1645] Step 10:

[1646] The server adjusts the user interface display based on the recognized emotion, such as by displaying a screen offering additional help information or detailed explanations if the user expresses dissatisfaction.

[1647] Step 11:

[1648] If a user feels that the displayed information is incorrect, they can use the feedback function to send a correction request to the server. Based on the emotional data acquired by the emotion engine, the priority of the feedback content is determined and work is done to improve the system as necessary.

[1649] Step 12:

[1650] The server receives and analyzes the feedback data sent by users. Based on the analysis results, it retrains the system and image recognition model, updates the database, and aims to improve the accuracy of the entire system.

[1651] In this way, the system of the present invention improves the efficiency and accuracy of base station construction work through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[1652] Example 2

[1653] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1654] In conventional systems, image data analysis and user emotion recognition are often processed separately, resulting in insufficient information integration, reduced analysis accuracy, and difficulty in effectively incorporating user feedback. Furthermore, while base station construction work requires the rapid identification of various components and the provision of related information, there has been a lack of systems that can address this need. The present invention aims to solve these problems by providing an efficient system that integrates image data analysis, user emotion recognition, and feedback collection and reflection.

[1655] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1656] In this invention, the server includes means for preprocessing the image data again and extracting features, means for identifying a product name based on the extracted features, and means for the server to search a database based on the identified product name and obtain corresponding product information. This allows for efficient integration of multiple processing steps, providing immediate feedback to the user, and enabling system adjustment based on the user's emotions.

[1657] "Means for acquiring image data" refers to a device or method for collecting visual information as digital data using an input device such as a camera or scanner.

[1658] "Preprocessing means" refers to a method or device that performs operations such as resizing, noise removal, and color space conversion on acquired image data before analysis, thereby preparing the data in a form suitable for analysis.

[1659] The "means for transmitting to a server" refers to a device or method for transferring image data processed by a terminal to a remote server via a network.

[1660] A "means for extracting features" is a method or device that identifies important patterns and attributes such as shape, color, and texture from image data and extracts them as quantities.

[1661] The "means for identifying product names" refers to a method or device that compares the extracted features with a known database and identifies the product or object that best matches.

[1662] "Means for searching a database and obtaining relevant product information" refers to a method or device for searching a database using a specified product name or other key and extracting related information (product name, specifications, installation method, etc.).

[1663] The "means for transmitting the acquired product name and related information from the server to the terminal" refers to a device or method for transferring the information searched and acquired by the server to the terminal via the network.

[1664] "Means for displaying on a user interface" refers to a device or method for visually displaying acquired information on a terminal display or screen.

[1665] "Means for analyzing a user's emotions and adjusting the system's behavior based on those emotions" refers to a method or device that analyzes a user's facial expressions and voice to identify their emotional state, and flexibly changes the system's processing, such as changing the user interface or providing additional information, depending on those emotions.

[1666] To implement this invention, the following specific steps must be taken. This system combines image data analysis and user emotion recognition to streamline specific tasks. Specifically, it includes a series of tasks from acquiring image data, analyzing it to identifying product names, providing related information based on that information, and collecting feedback based on the user's emotions. This system consists of the following main components:

[1667] System Components

[1668] 1. Server:

[1669] The server acts as the main processing unit, preprocessing image data, extracting features, identifying product names, acquiring related information, and analyzing user emotion data. The server achieves this processing by receiving and processing HTTP requests using Python's Flask and running deep learning models using TensorFlow.

[1670] 2. Terminal:

[1671] The device is responsible for acquiring and preprocessing image data, sending the data to the server, displaying the results obtained from the server, and collecting user emotion data. The device is equipped with a camera and microphone, and uses Python's OpenCV to perform preprocessing such as image resizing, noise reduction, and color space conversion.

[1672] 3. User:

[1673] The user operates the device to take and upload image data, check the displayed results, and provide necessary feedback. The user's emotional data is analyzed by the emotion engine, and the system's behavior is adjusted accordingly.

[1674] 4. Emotion Engine:

[1675] The emotion engine recognizes emotions from the user's facial expressions and voice and captures them as data. For example, a model using Python's OpenCV or TensorFlow analyzes facial expression data entered by the user through the device's camera and recognizes the emotion (satisfaction, dissatisfaction, surprise, etc.). This data is used to adjust the system's display content and operations according to the user's emotions.

[1676] Specific examples of operation

[1677] The following example shows the specific operation of the system.

[1678] Example 1: Taking and uploading image data

[1679] The user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other component, selects the image in the application, and taps the "Upload Image" button. For example, the user takes a photo of a base station antenna and taps the "Upload Image" button.

[1680] Example 2: Preprocessing and sending image data

[1681] The device resizes the captured image data, removes noise, and converts the color space, and then sends the preprocessed image data to the server as an HTTP POST request.

[1682] Example 3: Receiving and analyzing image data

[1683] The server receives the image data sent from the device, performs preprocessing again (edge ​​detection and image normalization), and then uses a deep learning model to extract features from the image and identify the product name based on those features. For example, the server may identify an image of an antenna as "Model A123."

[1684] Example 4: Retrieving product names and related information

[1685] The server searches the database based on the identified product name and obtains the relevant product information (product name, specifications, installation method, etc.).

[1686] Example 5: Returning and displaying results

[1687] The server encodes the acquired product information in JSON format and sends it to the device as an HTTP response. The device receives this, decodes it, and displays it in the user interface. For example, it might say, "This antenna is model A123. The specifications are as follows: ..."

[1688] Example 6: Emotion Recognition and Feedback

[1689] The emotion engine analyzes the user's facial expressions and voice to recognize emotions, which the server receives and adjusts the system's behavior. For example, if the user makes a dissatisfied face, the emotion engine will recognize this and display additional information or support options.

[1690] Example 7: Feedback collection and analysis

[1691] If the displayed information is incorrect, the user can use the feedback feature to send a correction request to the server, which analyzes the feedback and uses it to retrain the system and update the database.

[1692] Prompt Sentence Examples

[1693] "This antenna is model A123. Its specifications are as follows: ..."

[1694] Please provide feedback:

[1695] "Analysis results from the emotion engine: The user's emotion was recognized as dissatisfied. Additional supporting information is displayed."

[1696] As described above, the present invention provides an integrated system that includes everything from analyzing image data to recognizing user emotions and reflecting feedback, thereby improving work efficiency and accuracy.

[1697] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1698] Step 1:

[1699] Image capture and upload

[1700] Input: User instruction to take an image

[1701] How it works: A user uses the device's camera to take a photo of a radio, antenna, concrete pole, or other object. The user launches the application, selects the camera function, frames the object, and taps the capture button.

[1702] Output: Image data stored in the device

[1703] Step 2:

[1704] Select and upload an image

[1705] Input: Image data stored on the device

[1706] How it works: The user selects an image they have taken in the application and taps the "Upload Image" button. This action causes the device to temporarily store the selected image data and prepare it for transmission to the server.

[1707] Output: Image data ready to send to the server

[1708] Step 3:

[1709] Image data preprocessing

[1710] Input: Selected image data

[1711] What it does: The device preprocesses the image data, using Python's OpenCV to resize the image (e.g., change the resolution to 1024x768), denoise it (apply a Gaussian filter), and perform color space conversion (from RGB to grayscale).

[1712] Output: Preprocessed image data

[1713] Step 4:

[1714] Sending data to the server

[1715] Input: Preprocessed image data

[1716] Operation: The device sends the preprocessed image data to the server as an HTTP POST request, which includes a unique identifier (e.g., a UUID).

[1717] Output: Image data received by the server

[1718] Step 5:

[1719] Re-preprocessing of image data

[1720] Input: Image data received by the server

[1721] Operation: The server performs preprocessing on the received image data again, specifically edge detection (using the Canny algorithm) and image normalization (adjusting values ​​to the 0-1 range).

[1722] Output: Re-preprocessed image data

[1723] Step 6:

[1724] Image analysis and feature extraction

[1725] Input: Re-preprocessed image data

[1726] How it works: The server inputs image data into a deep learning model using TensorFlow and extracts features, such as shapes and patterns from the image using a convolutional neural network (CNN).

[1727] Output: Features extracted from the image

[1728] Step 7:

[1729] Identifying the product name

[1730] Input: extracted features

[1731] How it works: The server searches the database based on the extracted features to identify the best-matching product name. Specifically, it compares the features with known database entries and identifies the entries with the highest similarity.

[1732] Output: Identified product name

[1733] Step 8:

[1734] Obtaining product information

[1735] Input: Identified product name

[1736] Operation: The server searches the database based on the identified product name and retrieves the relevant product information (product name, specifications, installation method, etc.). The database search uses SQL.

[1737] Output: Retrieved product information

[1738] Step 9:

[1739] Encoding and sending the results

[1740] Input: Retrieved product information

[1741] Operation: The server encodes the acquired product information in JSON format and sends it to the terminal as an HTTP response.

[1742] Output: Product information sent to the terminal

[1743] Step 10:

[1744] Displaying the results

[1745] Input: Product information sent from the server

[1746] Operation: The device decodes the received JSON data and displays it in the user interface. The user can view product information on the display. For example, "This antenna is model A123. Specifications are as follows: ..."

[1747] Output: Product information displayed in the user interface

[1748] Step 11:

[1749] Emotion recognition and feedback collection

[1750] Input: User's facial expression data and voice data

[1751] How it works: The emotion engine uses the device's camera and microphone to collect the user's facial expressions and voice. It uses Python's OpenCV and TensorFlow to analyze the facial expressions and voice in real time and recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives the emotion data and adjusts the system's display and operations based on the emotion.

[1752] Output: Recognized user emotion data and the system's display content adjusted accordingly

[1753] Step 12:

[1754] Analyzing feedback and improving the system

[1755] Input: User-provided feedback and emotional data

[1756] How it works: The user taps the "Feedback" button to report problems with the displayed information and submit suggestions for correction. The server uses the emotion data to prioritize the feedback and use it to retrain the system and update the database.

[1757] Output: Improved system and updated database

[1758] Through the above processing steps, the system efficiently and accurately analyzes image data, recognizes user emotions, and collects and reflects feedback.

[1759] (Application example 2)

[1760] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1761] In recent years, quality control on factory production lines has become increasingly important, but traditional quality inspections often rely on human labor, resulting in many issues in terms of efficiency and accuracy. Furthermore, there is also the problem that feedback based on the emotions and satisfaction of inspectors is not sufficiently provided, making it difficult to improve the work environment. In response to these issues, the challenge is to provide a more efficient and highly accurate quality inspection system by utilizing image data analysis and emotion recognition.

[1762] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for processing the image data, means for extracting features from the processed image data, means for identifying a product name based on the extracted features, means for acquiring the identified product name from a database, means for displaying the acquired product name and related information, and means for recognizing the user's emotions and adjusting the behavior of the system. This makes it possible to improve the efficiency and accuracy of quality inspections on production lines and provide feedback based on the emotions of workers.

[1763] "Image data" refers to visual information captured using a camera or other imaging device.

[1764] "Processing means" refers to a method or device that performs preprocessing such as resizing, noise removal, and color space conversion on the acquired image data to make it suitable for analysis.

[1765] "Features" are attributes such as specific patterns, shapes, colors, and textures extracted from image data, and are information that enables product identification and classification.

[1766] "Product name" refers to the product's unique name or model number that is searched for in the database based on the identified features.

[1767] A "database" is an information collection system that stores product names and other related information based on identified features and allows them to be searched and retrieved.

[1768] A "user interface" is a screen or operation panel that allows the user to exchange information with the system, and is capable of displaying and inputting information.

[1769] The "means for recognizing emotions" refers to a method or device for analyzing emotions from a user's facial expressions and voice and identifying their emotional state.

[1770] The "means for adjusting the system's behavior" refers to a method or device for dynamically changing the system's operation or display content based on the results of emotion recognition.

[1771] The following specific steps must be taken to implement this invention. The present invention provides a system for improving the efficiency of quality control on factory production lines, combining image data analysis with user emotion recognition. This system is primarily composed of a server, a terminal, a user, and an emotion engine.

[1772] First, the user takes a picture of the product using the device's camera. After taking the picture, the user selects the image in the application and taps the "Upload Image" button. This selects the image data in the device. For example, the user takes a picture of a product on a production line and taps the "Upload Image" button.

[1773] Next, the device receives the selected image data and performs preprocessing. Specifically, it resizes the image, removes noise, converts color space, and prepares it for analysis. This preprocessing improves the accuracy of analysis on the server. The preprocessed image data is then sent to the server as an HTTP POST request.

[1774] The server receives image data sent from the device. After receiving the data, the server performs preprocessing again (for example, edge detection and image normalization) to make it suitable for analysis. The preprocessed image data is input into an image recognition model. The image recognition model uses a deep learning algorithm with TensorFlow to extract features from the image and identify similar product names based on the extracted features. As a specific example, the server analyzes an uploaded product image and identifies that the image is of product "model number 123."

[1775] Next, the server searches a database based on the identified product name and retrieves the corresponding product information (product name, specifications, installation method, etc.). The retrieved product information provides all the information the user needs. The server encodes the retrieved product information in JSON format and sends it to the terminal as an HTTP response. The terminal receives this response, decodes it, and displays it on the user interface. The user can check the displayed product information. As a specific example, the server encodes the retrieved product information for "model number 123" in JSON format and sends it back to the terminal. The terminal receives this and displays, "This product is model number 123. Its specifications are as follows: ..."

[1776] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions (e.g., satisfaction, dissatisfaction, surprise, etc.). The server receives this emotion data and adjusts the system's behavior based on the user's emotions. Specifically, it can change the content displayed in the user interface or provide additional help information. For example, if the user makes a face expressing dissatisfaction with the results, the emotion engine could recognize this and display more detailed information or additional support options.

[1777] For example, the prompt for taking a picture is:

[1778] "Take a photo of the defect in the product and upload the image. We'll find out what the defect is."

[1779] In this way, the system of the present invention can improve the efficiency and accuracy of quality control on factory production lines through a series of processes, from capturing image data to analyzing it, displaying the results, and collecting feedback that integrates user emotion recognition.

[1780] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1781] Step 1:

[1782] A user takes a picture of a product using the device's camera. The input is a physical image of the product, and the output is digital image data stored on the device. Specifically, the user opens an application on the device, uses the camera function to take a picture of the product, and saves the image in the application.

[1783] Step 2:

[1784] The user selects an image in the application and taps the "Upload Image" button. The input is the image data saved in step 1, and the output is the unprocessed image data ready to be sent from the device application to the server. Specifically, the user selects an image file in the application and taps the "Upload Image" button.

[1785] Step 3:

[1786] The device receives the selected image data and performs preprocessing. The input is the image data selected by the user, and the output is the preprocessed image data. Specifically, the device performs image resizing, noise removal, color space conversion, etc., to prepare the image for analysis. OpenCV's resize function is used for resizing, and blur or GaussianBlur functions are used for noise removal.

[1787] Step 4:

[1788] The preprocessed image data is sent to the server as an HTTP POST request. The input is the preprocessed image data, and the output is the data to be sent to the server. In specific operations, the terminal generates an HTTP POST request and sends the request including the preprocessed image data to the server.

[1789] Step 5:

[1790] The server receives the image data sent from the device and preprocesses it again. The input is the preprocessed image data sent from the device, and the output is image data suitable for analysis. Specifically, the server performs edge detection and image normalization, preparing it for input into the TensorFlow deep learning model.

[1791] Step 6:

[1792] The server uses a deep learning model to extract features from images and identify product names based on the identified features. The input is preprocessed image data, and the output is the identified product name. Specifically, the server uses TensorFlow to analyze images, detect specific features, and identify product names based on them.

[1793] Step 7:

[1794] The server searches the database based on the identified product name and retrieves the corresponding product information. The input is the identified product name and the output is the retrieved product information. Specifically, the server executes a database query to retrieve information such as the name, specifications, and installation method of the corresponding product.

[1795] Step 8:

[1796] The product information acquired by the server is encoded in JSON format and sent to the terminal as an HTTP response. The input is the acquired product information, and the output is JSON format data. Specifically, the server converts the acquired product information into JSON and sends it to the terminal as an HTTP response.

[1797] Step 9:

[1798] The terminal receives the response from the server, decodes it, and displays it on the user interface. The input is the JSON data returned from the server, and the output is the product information displayed on the user interface. Specifically, the terminal parses the JSON data and displays the product information on the screen.

[1799] Step 10:

[1800] The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions. The input is the user's facial expressions and voice data, and the output is the recognized emotion data. Specifically, the device or server analyzes the user's emotions using Haar Cascade or other emotion recognition algorithms.

[1801] Step 11:

[1802] The server adjusts the system behavior based on the emotion recognition results. The input is the recognized emotion data, and the output is the adjusted system behavior or additional display information. Specific actions include changing the display content of the user interface or providing additional help information.

[1803] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1804] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1805] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1806] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1807] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1808] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1809] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1810] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1811] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1812] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1813] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1814] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1815] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1816] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1817] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1818] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1819] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1820] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1821] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1822] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1823] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1824] The following is further disclosed regarding the above embodiment.

[1825] (Claim 1)

[1826] means for acquiring image data;

[1827] means for processing the image data;

[1828] means for extracting features from the processed image data;

[1829] means for identifying a product name based on the extracted feature amount;

[1830] means for acquiring the identified product name from a database;

[1831] The system includes means for displaying the obtained product name and related information.

[1832] (Claim 2)

[1833] The system of claim 1 , further comprising means for pre-processing the image data.

[1834] (Claim 3)

[1835] The system of claim 1 , further comprising: means for displaying the product name and related information in a user interface.

[1836] "Example 1"

[1837] (Claim 1)

[1838] means for acquiring image data;

[1839] means for preprocessing the image data;

[1840] means for transmitting the preprocessed image data to a server;

[1841] means for further processing the image data received by the server;

[1842] means for extracting features from the processed image data;

[1843] means for identifying a product name based on the extracted feature amount;

[1844] means for acquiring the specified product name and related information from a database;

[1845] means for encoding and transmitting the acquired product name and related information;

[1846] means for decoding and displaying said transmitted information.

[1847] (Claim 2)

[1848] 10. The system of claim 1, further comprising means for receiving and analyzing feedback from the user.

[1849] (Claim 3)

[1850] The system of claim 1 , further comprising: means for displaying the product name and related information in a user interface.

[1851] "Application Example 1"

[1852] (Claim 1)

[1853] means for acquiring image data;

[1854] means for processing the image data;

[1855] means for extracting features from the processed image data;

[1856] means for identifying an item name based on the extracted feature amount;

[1857] means for acquiring the identified item name from a database;

[1858] means for displaying the acquired item name and related information;

[1859] a means for acquiring and transmitting component data using an image acquisition device mounted on the factory machine;

[1860] means for analyzing the transmitted data and automatically returning the item name and related information;

[1861] A system including:

[1862] (Claim 2)

[1863] means for preprocessing the image data;

[1864] 10. The system of claim 1.

[1865] (Claim 3)

[1866] means for displaying the item name and related information on a user interface;

[1867] 10. The system of claim 1.

[1868] "Example 2: Combining Emotion Engines"

[1869] (Claim 1)

[1870] means for acquiring image data;

[1871] means for preprocessing the image data;

[1872] means for transmitting the preprocessed image data to a server;

[1873] means for preprocessing the image data again and extracting features;

[1874] means for identifying a product name based on the extracted feature amount;

[1875] a means for the server to search a database based on the identified product name and acquire corresponding product information;

[1876] means for transmitting the acquired product name and related information from a server to a terminal;

[1877] means for displaying the acquired product name and related information on a user interface of a terminal;

[1878] A system that includes means for analyzing a user's emotions and adjusting the system's behavior based on those emotions.

[1879] (Claim 2)

[1880] The system of claim 1 , further comprising means for pre-processing the image data received by the server.

[1881] (Claim 3)

[1882] 10. The system of claim 1, further comprising means for collecting feedback entered by the user and using the feedback to improve the system.

[1883] "Application example 2 when combining emotion engines"

[1884] (Claim 1)

[1885] means for acquiring image data;

[1886] means for processing the image data;

[1887] means for extracting features from the processed image data;

[1888] means for identifying a product name based on the extracted feature amount;

[1889] means for acquiring the identified product name from a database;

[1890] a means for displaying the acquired product name and related information;

[1891] means for recognizing user emotions and adjusting system behavior;

[1892] A system including:

[1893] (Claim 2)

[1894] The system of claim 1 , further comprising means for pre-processing the image data.

[1895] (Claim 3)

[1896] The system of claim 1 , further comprising: means for displaying the product name and related information in a user interface. [Explanation of symbols]

[1897] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring image data; means for processing the image data; means for extracting features from the processed image data; means for identifying a product name based on the extracted feature amount; means for acquiring the identified product name from a database; The system includes means for displaying the obtained product name and related information.

2. The system of claim 1 further comprising means for preprocessing the image data.

3. The system of claim 1 , further comprising: means for displaying the product name and related information in a user interface.

Images