system

Abstract

We provide the system. [Solution] Image processing means for preprocessing image data acquired by a video acquisition device to identify pests, A means for producing an optimal chemical substance based on the identified type of pest, A means for dispersing the generated chemical substance to a designated location, A data management means for accumulating and analyzing identification results and dispersal history, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In agriculture, the damage to crops caused by pests is significant, and the adverse effects on the human body and environmental burden caused by the use of chemical pesticides have become problems. Conventional pheromone traps need to be set individually, and there is a problem of lack of efficiency. Therefore, it is an important issue to automate pest detection to optimal pheromone generation and spraying and provide a safe and effective pest management method.

Means for Solving the Problems

[0005] This invention provides a system that includes image processing means for preprocessing image data acquired by an image acquisition device to identify pests. Furthermore, it includes generation means for generating an optimal chemical substance based on the identified type of pest, and spraying means for spraying the generated chemical substance to a predetermined location. This enables attraction using pheromones optimized for each pest, making it possible to achieve efficient pest control while reducing the use of pesticides. In addition, it includes data management means for accumulating and analyzing identification results and spraying history, contributing to improved efficiency and accuracy in agricultural work.

[0006] An "image acquisition device" is a device used to acquire video data from the environment, and generally refers to a camera device.

[0007] "Image processing means" refers to a process or apparatus for analyzing acquired video data and performing pre-processing such as noise reduction and contrast adjustment.

[0008] "Pests" refer to animals that cause damage to agricultural crops and are organisms that require management or removal.

[0009] "Identification" refers to the act or process of distinguishing a specific object from among multiple objects.

[0010] A "chemical substance" is a substance with a specific chemical composition, and in this context, it refers to attractants such as pheromones.

[0011] "Means of production" refers to an apparatus or process for synthesizing or manufacturing a specific chemical substance.

[0012] "Dispersion means" refers to a device or process for applying or distributing the generated chemical substance into the environment.

[0013] "Data management means" refers to a process or system for accumulating acquired data and results for later analysis and utilization.

[0014] "Accumulation" refers to an act or method for long-term storage of data or information.

[0015] "Analysis" refers to a process for detailed investigation of data or information to understand its content and characteristics.

Brief Description of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [[ID=�4]] [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be described.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] As an embodiment for carrying out the present invention, a specific implementation example of a pest control system will be described.

[0038] System Overview

[0039] This system consists of multiple functional modules, primarily performing functions such as video acquisition, image processing, chemical generation, spraying, and data management. The system comprises servers, terminals, and users, each fulfilling its respective role to effectively suppress pest damage.

[0040] Program processing

[0041] Video acquisition and preprocessing

[0042] The server acquires real-time video data from network cameras installed on the farm. This provides a foundation for continuously monitoring pest outbreaks. To efficiently process the video data, the server divides it into frames and pre-processes it to reduce noise as needed.

[0043] Pest identification

[0044] The server uses pre-processed frames to apply a convolutional neural network (CNN), an AI model, to identify specific pests. In this process, each identified pest is assigned a specific label, and processing proceeds based on that label.

[0045] Pheromone production and dispersal

[0046] The terminal generates the optimal pheromone based on pest information identified by the server. By dynamically adjusting the mixing ratio of chemicals, it is possible to generate the most effective pheromone for specific pests. The generated pheromone is automatically moved by the terminal to the application location and appropriately dispersed using a spraying device.

[0047] Data management and analysis

[0048] The server stores data obtained from each process, preparing it for later analysis. This data includes the types and numbers of identified pests, and the amount of pheromones sprayed. Users can access this data to help predict damage and develop further countermeasures.

[0049] Specific example

[0050] When a user manages a cornfield, the server analyzes the video feed from the camera and identifies the corn borer. Instructions for generating the necessary pheromones are sent to the terminal, which then generates the pheromone best suited to the corn borer and sprays it in specific areas of the field. Because this process is optimized for each pest species, effective pest control is achieved without the use of pesticides.

[0051] This system is a useful solution that can significantly reduce crop damage caused by pests while also improving the efficiency of agricultural work.

[0052] The following describes the processing flow.

[0053] Step 1:

[0054] The server acquires video in real time through the installed cameras. The video is received as a stream and divided into individual images at a constant frame rate. The server temporarily stores this video data in a database and prepares it for subsequent processing.

[0055] Step 2:

[0056] The server preprocesses the acquired video frames using image processing algorithms. Specifically, filtering is performed to reduce noise, and brightness and contrast are adjusted. This improves the accuracy of pest identification.

[0057] Step 3:

[0058] The server analyzes pre-processed frames using a deep learning model to identify pests. The model is a convolutional neural network (CNN) that has already learned the characteristics of pests. The server labels the identified pests and determines their species and number.

[0059] Step 4:

[0060] The server determines the type and quantity of chemical substances to be produced based on the identification results and transmits this information to the terminal. The instructions from the server include which chemical substances to produce and in what quantities.

[0061] Step 5:

[0062] The terminal generates the appropriate chemical substances based on instructions from the server. If necessary, the terminal mixes the chemical components to create a pheromone in the optimal ratio.

[0063] Step 6:

[0064] The terminal transfers the generated pheromones to a dispersal device and performs dispersal towards the designated area. The dispersal range and duration follow a pre-configured profile.

[0065] Step 7:

[0066] The server comprehensively manages data after the entire process is complete. The server records analysis results and generation / spraying history, which can be used for future analysis and improvement. This further promotes the efficiency of pest management.

[0067] (Example 1)

[0068] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0069] Conventional pest management systems have struggled to effectively control pest damage to crops due to insufficient pest identification accuracy and inefficient chemical application. Furthermore, there has been a lack of integrated system design encompassing identification, generation, application, and management, leaving room for improvement in overall operational efficiency. To address these challenges, more accurate identification functions and efficient chemical generation and application technologies are required.

[0070] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0071] In this invention, the server includes an image analysis means for preprocessing visual data acquired by an image acquisition device and identifying pests, a synthesis means for generating an optimal chemical substance based on the identified type of pest, and a distribution means for spraying the generated chemical substance to a predetermined location. This enables real-time identification of pests, generation of the optimal chemical substance based on the results, and efficient spraying.

[0072] A "video acquisition device" is a device used to collect visual data and is used to acquire images of the environment in real time.

[0073] "Visual data" refers to image information collected by video acquisition devices, and by analyzing this data, it becomes possible to identify pests.

[0074] "Preprocessing" refers to the initial stages of processing that include denoising and adjusting the data to improve the quality of the raw data and facilitate analysis.

[0075] "Pests" refer to insects that are harmful to crops, and the goal is to improve agricultural productivity by controlling their occurrence.

[0076] "Image analysis means" refers to technical means for identifying and classifying objects based on visual data, and it is common to use machine learning algorithms.

[0077] A "convolutional neural network" is a type of deep learning specifically designed for image analysis, enabling it to automatically learn and identify features within images.

[0078] A "synthesis means" is a technical means for compounding chemical substances based on identified targets, and allows for dynamic changes in the mixing ratio.

[0079] "Chemical substances" refer to substances produced for pest control, which, when formulated appropriately, act effectively against specific pests.

[0080] A "distribution mechanism" is a system for delivering and dispersing the generated chemical substances to a designated location, and plays a role in efficiently delivering substances to the target location.

[0081] An "information management system" is a system mechanism for accumulating collected data and using it for analysis and future decision-making.

[0082] One embodiment of this invention is the construction of a comprehensive pest control system. The main components of the system are a server, terminals, and users.

[0083] The server first uses network cameras as video acquisition devices to acquire visual data in real time from monitored areas such as farms. The acquired visual data is preprocessed internally to remove noise. This makes the visual data suitable for analysis.

[0084] The server uses pre-processed visual data and employs convolutional neural network (CNN) technology to identify pests. This allows for the identification of pest types and numbers, providing crucial information for proceeding to the next step.

[0085] The terminal generates chemical substances based on pest information identified by the server. A dynamically adjustable chemical synthesis device is used for this generation. The synthesized chemical substances are prepared in optimal proportions for the target pest and used in the form of pheromones or other substances.

[0086] The terminal also features a dispensing device for spraying the generated chemicals. This dispensing device can be mounted on drones or autonomous vehicles, allowing for precise delivery to designated spraying points and efficient dispensing.

[0087] Users can access identification data and distribution history provided by the server to check the situation in real time and devise countermeasures as needed. Furthermore, this data will be used to design future countermeasures and improve the system.

[0088] For example, if a user manages a cornfield, the server analyzes visual data from network cameras to identify corn borers. The terminal can then generate the appropriate pheromone and spray it on specific areas of the field. This process enables effective pest control without the use of pesticides.

[0089] An example of a prompt would be, "Please propose the optimal method for generating and distributing pheromones as a pest control measure in a cornfield." This would then generate specific countermeasures using a generative AI model.

[0090] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0091] Step 1:

[0092] The server acquires visual data in real time from network cameras. The input is video footage of the farm captured by the cameras. The server divides this video data frame by frame and applies a noise reduction filter to obtain clear image data as output. Specific operations include dimensionality reduction and removal of unnecessary data.

[0093] Step 2:

[0094] The server identifies pests using a convolutional neural network (CNN) based on pre-processed visual data. The input is pre-processed image frames. The server applies the CNN to recognize and label specific pests from each frame, outputting identified pest information. This process involves feature extraction and pattern matching to effectively classify pests.

[0095] Step 3:

[0096] The terminal generates chemical substances based on pest information output from the server. The input is data regarding the type and number of pests. Using a synthesis device employing dynamic mixing ratios, the terminal produces the optimal chemical substance for a specific pest as output. Specific operations include weighing and mixing chemical components.

[0097] Step 4:

[0098] The terminal disperses the generated chemical substance into a designated area. The input is the generated chemical substance. The terminal controls the dispersal device and performs actions to effectively distribute the chemical substance towards the target area. By using automated vehicles or drones, precise and efficient dispersal is achieved as the output.

[0099] Step 5:

[0100] The server stores data related to identification and distribution, preparing it for later analysis. Input consists of data obtained from all processing steps. The server stores the information in a database and provides analysis results as output data, which can be used for damage prediction and future countermeasure planning. Specific operations include data integration and information visualization.

[0101] (Application Example 1)

[0102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0103] In industrial production facilities, it is crucial to protect products and equipment from pests while maintaining a hygienic environment. However, conventional methods require regular human inspections, which are time-consuming and labor-intensive, and make it difficult to address pest infestations in real time. Therefore, there is a need for a system that can autonomously detect pests and respond immediately.

[0104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0105] In this invention, the server includes an image processing means for preprocessing image data acquired by a video acquisition device and identifying pests; a generation means for generating an optimal chemical substance based on the identified type of pest; a dispensing means for dispensing the generated chemical substance to a predetermined location; and a display means for monitoring the occurrence of pests within an industrial production facility using an autonomous mobile device and displaying pest occurrence information in real time. This enables real-time monitoring within the facility and rapid pest control.

[0106] A "video acquisition device" is a device used to acquire video data in real time within an industrial production facility.

[0107] "Image processing means" refers to a technique used to preprocess acquired video data and identify pests.

[0108] "Generation method" refers to a technology for generating the optimal chemical substance according to the identified type of pest.

[0109] "Dispersion methods" refer to techniques for effectively dispersing generated chemical substances to specific locations within industrial production facilities.

[0110] A "data management system" is a system for accumulating identification results and distribution history, and for analyzing them.

[0111] An "autonomous mobile device" is an automated device that moves around within an industrial production facility while monitoring for pest outbreaks.

[0112] "Display means" refers to a device or system for visually displaying pest outbreak information in real time.

[0113] The system that realizes this application example is built with the aim of autonomously detecting pests in industrial production facilities and taking rapid and effective countermeasures. The program of this system is mainly based on the Robot Operating System (ROS) and uses software libraries such as OpenCV and TENSORFLOW®.

[0114] The server acquires video data in real time from video acquisition devices installed at the facility. This data is preprocessed using OpenCV, including noise reduction and frame splitting. Subsequently, a convolutional neural network (CNN) implemented with TensorFlow analyzes the video data and identifies pests. Based on the information of the identified pests, the autonomous mobile device utilizes generation methods to produce the optimal chemical substance and moves to the appropriate location.

[0115] The terminal automatically sprays chemicals generated at its destination. A dynamically adjustable mixing ratio is used for the generation of these chemicals, and they are appropriately distributed by the spraying mechanism. The data obtained during these processes is stored on a server and analyzed via SQLite for data management.

[0116] Users can use a display to check the pest infestation status within the facility in real time. Devices such as smart glasses and head-mounted displays can be used for this purpose. A specific prompt for this application example would be: "Identify the objects contained in this frame, and if cockroaches are detected, initiate the corresponding pheromone production and dispersal process."

[0117] This system enables rapid pest control and automated hygiene management in industrial production facilities. It's a solution that supports more efficient operations while maintaining optimal hygiene conditions within the facility at all times.

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The server acquires video data in real time from video acquisition devices installed in industrial production facilities. This video data is used as input, and noise reduction and frame splitting are performed using OpenCV. The output is processed, clean image data.

[0121] Step 2:

[0122] The server inputs preprocessed image data into a convolutional neural network (CNN) built with TensorFlow. The CNN extracts features from the image and performs data calculations to identify pests. The output is the type and location information of the identified pests.

[0123] Step 3:

[0124] Based on the pest type obtained in Step 2, the server uses a generation AI model to form prompts for generating the optimal chemical substance. This prompt is sent to the terminal as input. The output is a recipe for a chemical substance suitable for the specific pest.

[0125] Step 4:

[0126] The terminal dynamically adjusts the mixing ratio to produce chemical substances based on the chemical recipe received from the server. Specifically, this production process involves precisely measuring and mixing different chemicals. The resulting chemical substances are then output.

[0127] Step 5:

[0128] The terminal moves the generated chemical substance to a designated location and prepares it for dispersal using an autonomous mobile device. It moves to the specified position and operates the spraying device to effectively disperse the chemical substance. The output is the chemical substance dispersed in the target area.

[0129] Step 6:

[0130] The server stores all data obtained from all processes in an SQLite database and performs analysis as a data management tool. This includes organizing and analyzing identification results and distribution history. An analysis report accessible to the user is then generated.

[0131] Step 7:

[0132] Users can check pest outbreak information in real time using display devices. The analysis information generated in step 6 is displayed on smart glasses or a head-mounted display. The output is a visually displayed dashboard of the pest management status.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] As an embodiment of the present invention, a specific implementation example in which an emotion engine is incorporated into a pest management system will be described. The emotion engine recognizes the user's emotional state and adjusts the system's operation based on that state.

[0135] System Overview

[0136] This system provides a more flexible and ergonomic agricultural solution by integrating pest control functions with user emotion recognition capabilities. In addition to conventional pest control functions, it adjusts the parameters of chemical generation and spraying according to the user's emotional state, enabling operation that combines safety and efficiency.

[0137] Program processing

[0138] Video acquisition and preprocessing

[0139] The server acquires video footage from the installed cameras. This video data is used for pest detection and as input data for the emotion engine. The server divides the image data frame by frame and performs noise reduction.

[0140] Pest identification

[0141] The server utilizes deep learning models to identify pests from pre-processed frames. This allows for rapid and accurate identification of pest types and distributions, providing foundational data for appropriate responses.

[0142] emotion recognition

[0143] The server analyzes the voice data provided by the user and processes it using an emotion engine. This process determines the user's emotional state, and this data influences subsequent decision-making processes.

[0144] Pheromone production and dispersal regulation

[0145] Based on emotional data obtained from the emotion engine, the device adjusts its pheromone generation and dispersal settings. For example, if the user is feeling stressed, the system increases the amount of pheromones produced to enable a quicker response.

[0146] Data management and proposal of improvement measures.

[0147] The server accumulates data from all processes and provides a follow-up function that suggests more efficient pest control methods through daily analysis. Users can utilize this data to select the optimal agricultural methods tailored to the environment and their emotions.

[0148] Specific example

[0149] When the emotion engine detects that a user is experiencing stress, the terminal will send a command from the server to spray a larger amount of pheromones in areas heavily damaged by corn borers. This enables rapid pest control and allows for agricultural management that is tailored to the user's emotional state.

[0150] This design reduces pest damage to crops, alleviates the mental burden on users, and enables efficient agricultural management.

[0151] The following describes the processing flow.

[0152] Step 1:

[0153] The server acquires video data in real time from network cameras installed on the farm. This video is then divided into frames and pre-processed to remove noise.

[0154] Step 2:

[0155] The server identifies specific pests by analyzing pre-processed frames using a deep learning convolutional neural network (CNN). At this stage, various pests, such as corn borers and spider mites, are classified and detected.

[0156] Step 3:

[0157] The user provides voice input to the system. The server analyzes this voice data using an emotion engine to determine the user's emotional state, such as joy, stress, or anxiety.

[0158] Step 4:

[0159] The server combines the results from the emotion engine and the pest detection results to determine the type and amount of pheromones to be produced. This information is then sent to the terminal as an instruction to produce pheromones.

[0160] Step 5:

[0161] The device prepares the necessary chemicals and generates pheromones based on instructions from the server. In particular, adjustments are made to increase the amount of pheromones produced when the user is experiencing stress.

[0162] Step 6:

[0163] The device transfers the generated pheromones to a dispersal device and sprays them in a specific area of ​​the designated farm. The timing and range of pheromone dispersal are fine-tuned based on the emotional state.

[0164] Step 7:

[0165] The server collects and stores data about the entire process and records the history of each processing step. Based on this data, users can receive suggestions for improving their future farming plans.

[0166] (Example 2)

[0167] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0168] Conventional pest management systems focus solely on pest identification and response, neglecting consideration for the user's emotional state. This can lead to a stressful user experience, hindering efficient and comfortable operation. The present invention aims to enable flexible system operation that takes the user's emotional state into account.

[0169] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0170] In this invention, the server includes an image analysis means for preprocessing visual information obtained by an image acquisition means and identifying pests; a generation means for generating an optimal chemical substance based on the identified type of pest; a spraying means for spraying the generated chemical substance to a predetermined location; an emotion recognition means for analyzing the user's emotions and adjusting system operation; and a data management means for accumulating and analyzing identification results, spraying history, and user emotion information and suggesting improvement measures. This enables efficient and comfortable pest management that responds to the user's emotional state.

[0171] "Image acquisition means" refers to equipment or devices installed to capture visual information, and which have the function of acquiring image data necessary for the system.

[0172] "Image analysis means" refers to a technology that has the function of identifying and analyzing a specific object from acquired visual information, and is used to determine the type and number of pests.

[0173] "Generation means" refers to a device or group of devices that has the function of producing the optimal chemical substance based on the identified target, and which can be adjusted according to dynamic conditions.

[0174] A "dispersion device" is a device that has a mechanism for distributing and spraying the generated chemical substance to a specific location, enabling spraying in a precise amount and over a wide area.

[0175] "Emotion recognition means" refers to a technology that has the function of determining the emotional state from the user's speech and actions, and adjusting the operation of the entire system based on the results.

[0176] A "data management system" is a system that stores identification results, distribution history, and user sentiment information, and analyzes and evaluates this data to suggest more efficient operational methods.

[0177] As an embodiment of this invention, the system is an integrated management device that enables pest management and flexible responses based on the user's emotional state. Specifically, the server uses visual information obtained in real time through video acquisition equipment and image analysis software to remove noise and identify pests. By using a neural network for image analysis, the types and distribution of pests can be accurately grasped.

[0178] The server further receives the user's voice input and analyzes the user's emotional state by utilizing a generative AI model with emotion recognition capabilities. This analysis is used by the device to dynamically adjust the amount of pheromones produced and dispersed. The generated pheromones are then effectively distributed to the necessary locations by appropriate dispersal equipment, minimizing the impact of pests.

[0179] For example, if the emotion engine detects that a user is feeling stressed, the system will automatically configure itself to spray more pheromones in areas where there is a high incidence of pests such as corn borers.

[0180] Furthermore, the server stores all identification results, pheromone dispersal history, and user sentiment information in a database and uses AI technology to perform daily analyses, presenting users with improved and more efficient pest control measures. This allows users to select agricultural methods based on the latest data.

[0181] An example of a prompt sentence for input to a generating AI model is, "Please explain the countermeasures to take when a user experiences stress while using the pest management system."

[0182] This configuration allows the system to achieve fully automated agricultural management that combines pest control and emotional considerations based on real-time data and user sentiment, contributing to improved productivity and user experience on the farm.

[0183] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0184] Step 1:

[0185] The server acquires real-time visual information using video acquisition equipment. This video data is used as input for the next analysis process. By dividing the acquired data frame by frame and performing image preprocessing to remove noise from each frame, a clear image is obtained. This clear image becomes important data for subsequent pest identification.

[0186] Step 2:

[0187] The server applies a deep learning-based neural network model to preprocessed image data as input. This model identifies the presence of pests and determines the type and location of each pest from each frame. The output provides detailed information about the identified pests. This information is used to determine appropriate countermeasures for each type of pest.

[0188] Step 3:

[0189] The user provides voice data via a dedicated voice input device. The server uses this voice data as input and performs emotion analysis of the user using an emotion engine that utilizes a generative AI model. The emotion engine analyzes the features of the voice and outputs the emotional state the user is feeling. This information about the determined emotional state is used to adjust the system's operation.

[0190] Step 4:

[0191] The device adjusts the pheromone generator based on the pest identification results and the user's emotional state. Based on the output from the emotion engine, it optimizes the amount of pheromone produced and the spraying pattern. Especially when the user is stressed, a rapid response is required, so the amount of pheromone produced is increased and the spraying target area is immediately updated. The output after spraying is the result of pheromone production and spraying.

[0192] Step 5:

[0193] The server stores all identification results, spraying history, and user sentiment data in a database. Using this data as input, it performs daily data analysis and proposes efficient pest management methods and optimal system operation based on user sentiment. The final output is an improvement report presented to the user, enabling them to make more informed decisions regarding future actions.

[0194] (Application Example 2)

[0195] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0196] In modern vehicle use, the emotional state of the driver significantly impacts the driving experience. However, conventional systems have difficulty providing flexible vehicle control in response to the driver's emotions, sometimes compromising a comfortable driving experience. Furthermore, the inability to maintain an appropriate in-vehicle environment leads to insufficient stress reduction for the driver. These challenges need to be addressed.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0198] In this invention, the server includes an image processing means for preprocessing image data acquired by a video acquisition device and identifying an object to be identified; a generation means for generating an optimal substance based on the type of identified object; a dispersal means for dispersing the generated substance to a predetermined location; and an emotion recognition means for analyzing the user's emotional state and adjusting the parameters of substance generation and dispersal based on that data. This enables flexible and comfortable control of the in-vehicle environment in accordance with the user's emotional state.

[0199] A "video acquisition device" is a device used to capture video data and provides the visual information that the system needs to process.

[0200] "Image processing means" refers to a function that performs a series of operations to analyze acquired image data and identify a specific object.

[0201] A "generating means" is a device or mechanism that dynamically produces the necessary substances or parameters based on the type of object identified.

[0202] "Dispersion means" refers to a mechanism or device that appropriately distributes or releases the generated substance to the target location.

[0203] "Data management means" refers to a function within a system that manages the process for accumulating and analyzing identification results and distribution history.

[0204] "Emotion recognition means" refers to a function that analyzes the user's emotional state and is responsible for the process of adjusting other system parameters based on that information.

[0205] In the system that implements this application example, the server uses video data obtained from a video acquisition device to perform image processing and identify the target object. This utilizes deep learning models such as convolutional neural networks. The server also uses audio data to perform emotion recognition. For emotion recognition, it uses audio analysis technology to determine the user's emotional state and adjusts the properties and dispersion parameters of the generated substance based on the analysis results.

[0206] As a concrete example, the server uses cameras and microphones inside the vehicle to detect the emotional state of the user from their facial expressions and voice data while they are riding. If the emotion recognition system determines that the user is experiencing stress, the system adjusts the in-vehicle environment. For example, it may change the air conditioning temperature or play relaxing music to improve the user's comfort.

[0207] In this process, the generative AI model used is input with the following prompt:

[0208] "If you are currently feeling stressed, adjust the air conditioning to a comfortable temperature and play a relaxing playlist to create a more comfortable environment in your car."

[0209] This configuration enables flexible and comfortable control of the in-car environment in response to the user's emotional state.

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The server acquires video and audio data from cameras and microphones installed inside the vehicle. The input is real-time video and audio, which are used directly in the next processing step.

[0213] Step 2:

[0214] The server divides the acquired video data into frames and performs noise reduction. Then, it uses a deep learning model (particularly a convolutional neural network) to perform face recognition and facial expression analysis to identify the user's emotions. The output of this step is emotion data estimated from the user's facial expressions.

[0215] Step 3:

[0216] The server analyzes the voice data using an emotion recognition engine to determine the user's emotions from the tone and pitch of the voice. The input is voice data, and the output is the analyzed emotional state.

[0217] Step 4:

[0218] The server integrates the emotion data obtained as outputs from steps 2 and 3 to determine the user's overall emotional state. Here, a weighted average of the information obtained from the video and audio is used to derive the user's emotion score. The output of this step is the user's overall emotional evaluation.

[0219] Step 5:

[0220] Based on the results of the emotion assessment, the server generates prompt messages to adjust the in-car environment using a generative AI model. For example, if "stress" is detected, instructions to lower the air conditioning temperature or play relaxing music will be generated. These prompt messages are generated automatically and executed in the following steps.

[0221] Step 6:

[0222] The terminal controls the in-car environment based on prompt messages received from the server. It performs actions such as adjusting the air conditioning and playing specified music through the speakers. The input is the prompt message, and the output is the adjusted in-car environment. This allows the user to comfortably enjoy an in-car space optimized for their emotional state.

[0223] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0224] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0225] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0226] [Second Embodiment]

[0227] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0228] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0229] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0230] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0231] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0232] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0233] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0234] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0235] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0236] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0237] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0238] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0239] As an embodiment for carrying out the present invention, a specific implementation example of a pest control system will be described.

[0240] System Overview

[0241] This system consists of multiple functional modules, primarily performing functions such as video acquisition, image processing, chemical generation, spraying, and data management. The system comprises servers, terminals, and users, each fulfilling its respective role to effectively suppress pest damage.

[0242] Program processing

[0243] Video acquisition and preprocessing

[0244] The server acquires real-time video data from network cameras installed on the farm. This provides a foundation for continuously monitoring pest outbreaks. To efficiently process the video data, the server divides it into frames and pre-processes it to reduce noise as needed.

[0245] Pest identification

[0246] The server uses pre-processed frames to apply a convolutional neural network (CNN), an AI model, to identify specific pests. In this process, each identified pest is assigned a specific label, and processing proceeds based on that label.

[0247] Pheromone production and dispersal

[0248] The terminal generates the optimal pheromone based on pest information identified by the server. By dynamically adjusting the mixing ratio of chemicals, it is possible to generate the most effective pheromone for specific pests. The generated pheromone is automatically moved by the terminal to the application location and appropriately dispersed using a spraying device.

[0249] Data management and analysis

[0250] The server stores data obtained from each process, preparing it for later analysis. This data includes the types and numbers of identified pests, and the amount of pheromones sprayed. Users can access this data to help predict damage and develop further countermeasures.

[0251] Specific example

[0252] When a user manages a cornfield, the server analyzes the video feed from the camera and identifies the corn borer. Instructions for generating the necessary pheromones are sent to the terminal, which then generates the pheromone best suited to the corn borer and sprays it in specific areas of the field. Because this process is optimized for each pest species, effective pest control is achieved without the use of pesticides.

[0253] This system is a useful solution that can significantly reduce crop damage caused by pests while also improving the efficiency of agricultural work.

[0254] The following describes the processing flow.

[0255] Step 1:

[0256] The server acquires video in real time through the installed cameras. The video is received as a stream and divided into individual images at a constant frame rate. The server temporarily stores this video data in a database and prepares it for subsequent processing.

[0257] Step 2:

[0258] The server preprocesses the acquired video frames using image processing algorithms. Specifically, filtering is performed to reduce noise, and brightness and contrast are adjusted. This improves the accuracy of pest identification.

[0259] Step 3:

[0260] The server analyzes pre-processed frames using a deep learning model to identify pests. The model is a convolutional neural network (CNN) that has already learned the characteristics of pests. The server labels the identified pests and determines their species and number.

[0261] Step 4:

[0262] The server determines the type and quantity of chemical substances to be produced based on the identification results and transmits this information to the terminal. The instructions from the server include which chemical substances to produce and in what quantities.

[0263] Step 5:

[0264] The terminal generates the appropriate chemical substances based on instructions from the server. If necessary, the terminal mixes the chemical components to create a pheromone in the optimal ratio.

[0265] Step 6:

[0266] The terminal transfers the generated pheromones to a dispersal device and performs dispersal towards the designated area. The dispersal range and duration follow a pre-configured profile.

[0267] Step 7:

[0268] The server comprehensively manages data after the entire process is complete. The server records analysis results and generation / spraying history, which can be used for future analysis and improvement. This further promotes the efficiency of pest management.

[0269] (Example 1)

[0270] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0271] Conventional pest management systems have struggled to effectively control pest damage to crops due to insufficient pest identification accuracy and inefficient chemical application. Furthermore, there has been a lack of integrated system design encompassing identification, generation, application, and management, leaving room for improvement in overall operational efficiency. To address these challenges, more accurate identification functions and efficient chemical generation and application technologies are required.

[0272] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0273] In this invention, the server includes an image analysis means for preprocessing visual data acquired by an image acquisition device and identifying pests, a synthesis means for generating an optimal chemical substance based on the identified type of pest, and a distribution means for spraying the generated chemical substance to a predetermined location. This enables real-time identification of pests, generation of the optimal chemical substance based on the results, and efficient spraying.

[0274] A "video acquisition device" is a device used to collect visual data and is used to acquire images of the environment in real time.

[0275] "Visual data" refers to image information collected by video acquisition devices, and by analyzing this data, it becomes possible to identify pests.

[0276] "Preprocessing" refers to the initial stages of processing that include denoising and adjusting the data to improve the quality of the raw data and facilitate analysis.

[0277] "Pests" refer to insects that are harmful to crops, and the goal is to improve agricultural productivity by controlling their occurrence.

[0278] "Image analysis means" refers to technical means for identifying and classifying objects based on visual data, and it is common to use machine learning algorithms.

[0279] A "convolutional neural network" is a type of deep learning specifically designed for image analysis, enabling it to automatically learn and identify features within images.

[0280] A "synthesis means" is a technical means for compounding chemical substances based on identified targets, and allows for dynamic changes in the mixing ratio.

[0281] "Chemical substance" refers to a substance produced for pest control, which effectively acts on specific pests through appropriate formulation.

[0282] "Distribution means" is a mechanism for delivering and spraying the generated chemical substance to a predetermined position, and plays a role in efficiently delivering the substance to the target location.

[0283] "Information management means" is a system mechanism for accumulating the collected data and utilizing it for analysis and future decision-making.

[0284] As a form of implementing this invention, a comprehensive pest management system is constructed. The main components of the system are a server, a terminal, and a user.

[0285] First, the server uses a network camera as a video acquisition device to obtain visual data in real time from a monitoring target area such as a farm. The acquired visual data is preprocessed internally to remove noise. As a result, the visual data is in a state suitable for analysis.

[0286] The server uses the preprocessed visual data and utilizes the technology of convolutional neural network (CNN) to identify pests. Thereby, the type and number of pests can be specified, and it is possible to obtain an important judgment material for proceeding to the next step.

[0287] Based on the pest information identified by the server, the terminal generates chemical substances. For generation, a dynamically adjustable chemical substance synthesizer is used. The synthesized chemical substance is made in an optimal formulation ratio for the target pests and is used in the form of pheromones, etc.

[0288] The terminal also has a distribution device for spraying the generated chemical substance. The distribution device can be mounted on a drone or an automobile, accurately transports to the set spraying points, and efficiently performs spraying.

[0289] Users can access identification data and distribution history provided by the server to check the situation in real time and devise countermeasures as needed. Furthermore, this data will be used to design future countermeasures and improve the system.

[0290] For example, if a user manages a cornfield, the server analyzes visual data from network cameras to identify corn borers. The terminal can then generate the appropriate pheromone and spray it on specific areas of the field. This process enables effective pest control without the use of pesticides.

[0291] An example of a prompt would be, "Please propose the optimal method for generating and distributing pheromones as a pest control measure in a cornfield." This would then generate specific countermeasures using a generative AI model.

[0292] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0293] Step 1:

[0294] The server acquires visual data in real time from network cameras. The input is video footage of the farm captured by the cameras. The server divides this video data frame by frame and applies a noise reduction filter to obtain clear image data as output. Specific operations include dimensionality reduction and removal of unnecessary data.

[0295] Step 2:

[0296] The server identifies pests using a convolutional neural network (CNN) based on pre-processed visual data. The input is pre-processed image frames. The server applies the CNN to recognize and label specific pests from each frame, outputting identified pest information. This process involves feature extraction and pattern matching to effectively classify pests.

[0297] Step 3:

[0298] The terminal generates chemical substances based on pest information output from the server. The input is data regarding the type and number of pests. Using a synthesis device employing dynamic mixing ratios, the terminal produces the optimal chemical substance for a specific pest as output. Specific operations include weighing and mixing chemical components.

[0299] Step 4:

[0300] The terminal disperses the generated chemical substance into a designated area. The input is the generated chemical substance. The terminal controls the dispersal device and performs actions to effectively distribute the chemical substance towards the target area. By using automated vehicles or drones, precise and efficient dispersal is achieved as the output.

[0301] Step 5:

[0302] The server stores data related to identification and distribution, preparing it for later analysis. Input consists of data obtained from all processing steps. The server stores the information in a database and provides analysis results as output data, which can be used for damage prediction and future countermeasure planning. Specific operations include data integration and information visualization.

[0303] (Application Example 1)

[0304] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0305] In industrial production facilities, it is crucial to protect products and equipment from pests while maintaining a hygienic environment. However, conventional methods require regular human inspections, which are time-consuming and labor-intensive, and make it difficult to address pest infestations in real time. Therefore, there is a need for a system that can autonomously detect pests and respond immediately.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0307] In this invention, the server preprocesses the image data acquired by the video acquisition device, and includes image processing means for identifying pests, generation means for generating an optimal chemical substance based on the identified pest type, spraying means for spraying the generated chemical substance to a predetermined location, and display means for monitoring the occurrence of pests using an autonomous mobile device and displaying the pest occurrence information in real time within an industrial production facility. Thereby, real-time monitoring within the facility and rapid pest control become possible.

[0308] The "video acquisition device" is a device for acquiring video data in real time within an industrial production facility.

[0309] The "image processing means" is a technology for preprocessing the acquired video data and used for identifying pests.

[0310] The "generation means" is a technology for generating an optimal chemical substance according to the identified pest type.

[0311] The "spraying means" is a technology for effectively spraying the generated chemical substance to a specific location within an industrial production facility.

[0312] The "data management means" is a system for accumulating and analyzing the identification results and spraying history.

[0313] The "autonomous mobile device" is an automated device for monitoring the occurrence of pests while moving within an industrial production facility.

[0314] The "display means" is a device or system for visually displaying the pest occurrence information in real time.

[0315] The system that realizes this application example is built with the aim of autonomously detecting pests in industrial production facilities and taking rapid and effective countermeasures. The program of this system is mainly based on the Robot Operating System (ROS) and uses software libraries such as OpenCV and TensorFlow.

[0316] The server acquires video data in real time from video acquisition devices installed at the facility. This data is preprocessed using OpenCV, including noise reduction and frame splitting. Subsequently, a convolutional neural network (CNN) implemented with TensorFlow analyzes the video data and identifies pests. Based on the information of the identified pests, the autonomous mobile device utilizes generation methods to produce the optimal chemical substance and moves to the appropriate location.

[0317] The terminal automatically sprays chemicals generated at its destination. A dynamically adjustable mixing ratio is used for the generation of these chemicals, and they are appropriately distributed by the spraying mechanism. The data obtained during these processes is stored on a server and analyzed via SQLite for data management.

[0318] Users can use a display to check the pest infestation status within the facility in real time. Devices such as smart glasses and head-mounted displays can be used for this purpose. A specific prompt for this application example would be: "Identify the objects contained in this frame, and if cockroaches are detected, initiate the corresponding pheromone production and dispersal process."

[0319] This system enables rapid pest control and automated hygiene management in industrial production facilities. It's a solution that supports more efficient operations while maintaining optimal hygiene conditions within the facility at all times.

[0320] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0321] Step 1:

[0322] The server acquires video data in real time from video acquisition devices installed in industrial production facilities. This video data is used as input, and noise reduction and frame splitting are performed using OpenCV. The output is processed, clean image data.

[0323] Step 2:

[0324] The server inputs preprocessed image data into a convolutional neural network (CNN) built with TensorFlow. The CNN extracts features from the image and performs data calculations to identify pests. The output is the type and location information of the identified pests.

[0325] Step 3:

[0326] Based on the pest type obtained in Step 2, the server uses a generation AI model to form prompts for generating the optimal chemical substance. This prompt is sent to the terminal as input. The output is a recipe for a chemical substance suitable for the specific pest.

[0327] Step 4:

[0328] The terminal dynamically adjusts the mixing ratio to produce chemical substances based on the chemical recipe received from the server. Specifically, this production process involves precisely measuring and mixing different chemicals. The resulting chemical substances are then output.

[0329] Step 5:

[0330] The terminal moves the generated chemical substance to a designated location and prepares it for dispersal using an autonomous mobile device. It moves to the specified position and operates the spraying device to effectively disperse the chemical substance. The output is the chemical substance dispersed in the target area.

[0331] Step 6:

[0332] The server stores all data obtained from all processes in an SQLite database and performs analysis as a data management tool. This includes organizing and analyzing identification results and distribution history. An analysis report accessible to the user is then generated.

[0333] Step 7:

[0334] Users can check pest outbreak information in real time using display devices. The analysis information generated in step 6 is displayed on smart glasses or a head-mounted display. The output is a visually displayed dashboard of the pest management status.

[0335] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0336] As an embodiment of the present invention, a specific implementation example in which an emotion engine is incorporated into a pest management system will be described. The emotion engine recognizes the user's emotional state and adjusts the system's operation based on that state.

[0337] System Overview

[0338] This system provides a more flexible and ergonomic agricultural solution by integrating pest control functions with user emotion recognition capabilities. In addition to conventional pest control functions, it adjusts the parameters of chemical generation and spraying according to the user's emotional state, enabling operation that combines safety and efficiency.

[0339] Program processing

[0340] Video acquisition and preprocessing

[0341] The server acquires video footage from the installed cameras. This video data is used for pest detection and as input data for the emotion engine. The server divides the image data frame by frame and performs noise reduction.

[0342] Pest identification

[0343] The server utilizes deep learning models to identify pests from pre-processed frames. This allows for rapid and accurate identification of pest types and distributions, providing foundational data for appropriate responses.

[0344] emotion recognition

[0345] The server analyzes the voice data provided by the user and processes it using an emotion engine. This process determines the user's emotional state, and this data influences subsequent decision-making processes.

[0346] Pheromone production and dispersal regulation

[0347] Based on emotional data obtained from the emotion engine, the device adjusts its pheromone generation and dispersal settings. For example, if the user is feeling stressed, the system increases the amount of pheromones produced to enable a quicker response.

[0348] Data management and proposal of improvement measures.

[0349] The server accumulates data from all processes and provides a follow-up function that suggests more efficient pest control methods through daily analysis. Users can utilize this data to select the optimal agricultural methods tailored to the environment and their emotions.

[0350] Specific example

[0351] When the emotion engine detects that a user is experiencing stress, the terminal will send a command from the server to spray a larger amount of pheromones in areas heavily damaged by corn borers. This enables rapid pest control and allows for agricultural management that is tailored to the user's emotional state.

[0352] This design reduces pest damage to crops, alleviates the mental burden on users, and enables efficient agricultural management.

[0353] The following describes the processing flow.

[0354] Step 1:

[0355] The server acquires video data in real time from network cameras installed on the farm. This video is then divided into frames and pre-processed to remove noise.

[0356] Step 2:

[0357] The server identifies specific pests by analyzing pre-processed frames using a deep learning convolutional neural network (CNN). At this stage, various pests, such as corn borers and spider mites, are classified and detected.

[0358] Step 3:

[0359] The user provides voice input to the system. The server analyzes this voice data using an emotion engine to determine the user's emotional state, such as joy, stress, or anxiety.

[0360] Step 4:

[0361] The server combines the results from the emotion engine and the pest detection results to determine the type and amount of pheromones to be produced. This information is then sent to the terminal as an instruction to produce pheromones.

[0362] Step 5:

[0363] The device prepares the necessary chemicals and generates pheromones based on instructions from the server. In particular, adjustments are made to increase the amount of pheromones produced when the user is experiencing stress.

[0364] Step 6:

[0365] The device transfers the generated pheromones to a dispersal device and sprays them in a specific area of ​​the designated farm. The timing and range of pheromone dispersal are fine-tuned based on the emotional state.

[0366] Step 7:

[0367] The server collects and stores data about the entire process and records the history of each processing step. Based on this data, users can receive suggestions for improving their future farming plans.

[0368] (Example 2)

[0369] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0370] Conventional pest management systems focus solely on pest identification and response, neglecting consideration for the user's emotional state. This can lead to a stressful user experience, hindering efficient and comfortable operation. The present invention aims to enable flexible system operation that takes the user's emotional state into account.

[0371] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0372] In this invention, the server includes an image analysis means for preprocessing visual information obtained by an image acquisition means and identifying pests; a generation means for generating an optimal chemical substance based on the identified type of pest; a spraying means for spraying the generated chemical substance to a predetermined location; an emotion recognition means for analyzing the user's emotions and adjusting system operation; and a data management means for accumulating and analyzing identification results, spraying history, and user emotion information and suggesting improvement measures. This enables efficient and comfortable pest management that responds to the user's emotional state.

[0373] "Image acquisition means" refers to equipment or devices installed to capture visual information, and which have the function of acquiring image data necessary for the system.

[0374] "Image analysis means" refers to a technology that has the function of identifying and analyzing a specific object from acquired visual information, and is used to determine the type and number of pests.

[0375] "Generation means" refers to a device or group of devices that has the function of producing the optimal chemical substance based on the identified target, and which can be adjusted according to dynamic conditions.

[0376] A "dispersion device" is a device that has a mechanism for distributing and spraying the generated chemical substance to a specific location, enabling spraying in a precise amount and over a wide area.

[0377] "Emotion recognition means" refers to a technology that has the function of determining the emotional state from the user's speech and actions, and adjusting the operation of the entire system based on the results.

[0378] A "data management system" is a system that stores identification results, distribution history, and user sentiment information, and analyzes and evaluates this data to suggest more efficient operational methods.

[0379] As an embodiment of this invention, the system is an integrated management device that enables pest management and flexible responses based on the user's emotional state. Specifically, the server uses visual information obtained in real time through video acquisition equipment and image analysis software to remove noise and identify pests. By using a neural network for image analysis, the types and distribution of pests can be accurately grasped.

[0380] The server further receives the user's voice input and analyzes the user's emotional state by utilizing a generative AI model with emotion recognition capabilities. This analysis is used by the device to dynamically adjust the amount of pheromones produced and dispersed. The generated pheromones are then effectively distributed to the necessary locations by appropriate dispersal equipment, minimizing the impact of pests.

[0381] For example, if the emotion engine detects that a user is feeling stressed, the system will automatically configure itself to spray more pheromones in areas where there is a high incidence of pests such as corn borers.

[0382] Furthermore, the server stores all identification results, pheromone dispersal history, and user sentiment information in a database and uses AI technology to perform daily analyses, presenting users with improved and more efficient pest control measures. This allows users to select agricultural methods based on the latest data.

[0383] An example of a prompt sentence for input to a generating AI model is, "Please explain the countermeasures to take when a user experiences stress while using the pest management system."

[0384] This configuration allows the system to achieve fully automated agricultural management that combines pest control and emotional considerations based on real-time data and user sentiment, contributing to improved productivity and user experience on the farm.

[0385] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0386] Step 1:

[0387] The server acquires real-time visual information using video acquisition equipment. This video data is used as input for the next analysis process. By dividing the acquired data frame by frame and performing image preprocessing to remove noise from each frame, a clear image is obtained. This clear image becomes important data for subsequent pest identification.

[0388] Step 2:

[0389] The server applies a deep learning-based neural network model to preprocessed image data as input. This model identifies the presence of pests and determines the type and location of each pest from each frame. The output provides detailed information about the identified pests. This information is used to determine appropriate countermeasures for each type of pest.

[0390] Step 3:

[0391] The user provides voice data via a dedicated voice input device. The server uses this voice data as input and performs emotion analysis of the user using an emotion engine that utilizes a generative AI model. The emotion engine analyzes the features of the voice and outputs the emotional state the user is feeling. This information about the determined emotional state is used to adjust the system's operation.

[0392] Step 4:

[0393] The device adjusts the pheromone generator based on the pest identification results and the user's emotional state. Based on the output from the emotion engine, it optimizes the amount of pheromone produced and the spraying pattern. Especially when the user is stressed, a rapid response is required, so the amount of pheromone produced is increased and the spraying target area is immediately updated. The output after spraying is the result of pheromone production and spraying.

[0394] Step 5:

[0395] The server stores all identification results, spraying history, and user sentiment data in a database. Using this data as input, it performs daily data analysis and proposes efficient pest management methods and optimal system operation based on user sentiment. The final output is an improvement report presented to the user, enabling them to make more informed decisions regarding future actions.

[0396] (Application Example 2)

[0397] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0398] In modern vehicle use, the emotional state of the driver significantly impacts the driving experience. However, conventional systems have difficulty providing flexible vehicle control in response to the driver's emotions, sometimes compromising a comfortable driving experience. Furthermore, the inability to maintain an appropriate in-vehicle environment leads to insufficient stress reduction for the driver. These challenges need to be addressed.

[0399] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0400] In this invention, the server includes an image processing means for preprocessing image data acquired by a video acquisition device and identifying an object to be identified; a generation means for generating an optimal substance based on the type of identified object; a dispersal means for dispersing the generated substance to a predetermined location; and an emotion recognition means for analyzing the user's emotional state and adjusting the parameters of substance generation and dispersal based on that data. This enables flexible and comfortable control of the in-vehicle environment in accordance with the user's emotional state.

[0401] A "video acquisition device" is a device used to capture video data and provides the visual information that the system needs to process.

[0402] "Image processing means" refers to a function that performs a series of operations to analyze acquired image data and identify a specific object.

[0403] A "generating means" is a device or mechanism that dynamically produces the necessary substances or parameters based on the type of object identified.

[0404] "Dispersion means" refers to a mechanism or device that appropriately distributes or releases the generated substance to the target location.

[0405] "Data management means" refers to a function within a system that manages the process for accumulating and analyzing identification results and distribution history.

[0406] "Emotion recognition means" refers to a function that analyzes the user's emotional state and is responsible for the process of adjusting other system parameters based on that information.

[0407] In the system that implements this application example, the server uses video data obtained from a video acquisition device to perform image processing and identify the target object. This utilizes deep learning models such as convolutional neural networks. The server also uses audio data to perform emotion recognition. For emotion recognition, it uses audio analysis technology to determine the user's emotional state and adjusts the properties and dispersion parameters of the generated substance based on the analysis results.

[0408] As a concrete example, the server uses cameras and microphones inside the vehicle to detect the emotional state of the user from their facial expressions and voice data while they are riding. If the emotion recognition system determines that the user is experiencing stress, the system adjusts the in-vehicle environment. For example, it may change the air conditioning temperature or play relaxing music to improve the user's comfort.

[0409] In this process, the generative AI model used is input with the following prompt:

[0410] "If you are currently feeling stressed, adjust the air conditioning to a comfortable temperature and play a relaxing playlist to create a more comfortable environment in your car."

[0411] This configuration enables flexible and comfortable control of the in-car environment in response to the user's emotional state.

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The server acquires video and audio data from cameras and microphones installed inside the vehicle. The input is real-time video and audio, which are used directly in the next processing step.

[0415] Step 2:

[0416] The server divides the acquired video data into frames and performs noise reduction. Then, it uses a deep learning model (particularly a convolutional neural network) to perform face recognition and facial expression analysis to identify the user's emotions. The output of this step is emotion data estimated from the user's facial expressions.

[0417] Step 3:

[0418] The server analyzes the voice data using an emotion recognition engine to determine the user's emotions from the tone and pitch of the voice. The input is voice data, and the output is the analyzed emotional state.

[0419] Step 4:

[0420] The server integrates the emotion data obtained as outputs from steps 2 and 3 to determine the user's overall emotional state. Here, a weighted average of the information obtained from the video and audio is used to derive the user's emotion score. The output of this step is the user's overall emotional evaluation.

[0421] Step 5:

[0422] Based on the results of the emotion assessment, the server generates prompt messages to adjust the in-car environment using a generative AI model. For example, if "stress" is detected, instructions to lower the air conditioning temperature or play relaxing music will be generated. These prompt messages are generated automatically and executed in the following steps.

[0423] Step 6:

[0424] The terminal controls the in-car environment based on prompt messages received from the server. It performs actions such as adjusting the air conditioning and playing specified music through the speakers. The input is the prompt message, and the output is the adjusted in-car environment. This allows the user to comfortably enjoy an in-car space optimized for their emotional state.

[0425] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0426] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0427] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0428] [Third Embodiment]

[0429] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0430] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0431] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0432] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0433] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0434] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0435] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0436] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0437] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0438] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0439] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0440] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0441] As an embodiment for carrying out the present invention, a specific implementation example of a pest control system will be described.

[0442] System Overview

[0443] This system consists of multiple functional modules, primarily performing functions such as video acquisition, image processing, chemical generation, spraying, and data management. The system comprises servers, terminals, and users, each fulfilling its respective role to effectively suppress pest damage.

[0444] Program processing

[0445] Video acquisition and preprocessing

[0446] The server acquires real-time video data from network cameras installed on the farm. This provides a foundation for continuously monitoring pest outbreaks. To efficiently process the video data, the server divides it into frames and pre-processes it to reduce noise as needed.

[0447] Pest identification

[0448] The server uses pre-processed frames to apply a convolutional neural network (CNN), an AI model, to identify specific pests. In this process, each identified pest is assigned a specific label, and processing proceeds based on that label.

[0449] Pheromone production and dispersal

[0450] The terminal generates the optimal pheromone based on pest information identified by the server. By dynamically adjusting the mixing ratio of chemicals, it is possible to generate the most effective pheromone for specific pests. The generated pheromone is automatically moved by the terminal to the application location and appropriately dispersed using a spraying device.

[0451] Data management and analysis

[0452] The server stores data obtained from each process, preparing it for later analysis. This data includes the types and numbers of identified pests, and the amount of pheromones sprayed. Users can access this data to help predict damage and develop further countermeasures.

[0453] Specific example

[0454] When a user manages a cornfield, the server analyzes the video feed from the camera and identifies the corn borer. Instructions for generating the necessary pheromones are sent to the terminal, which then generates the pheromone best suited to the corn borer and sprays it in specific areas of the field. Because this process is optimized for each pest species, effective pest control is achieved without the use of pesticides.

[0455] This system is a useful solution that can significantly reduce crop damage caused by pests while also improving the efficiency of agricultural work.

[0456] The following describes the processing flow.

[0457] Step 1:

[0458] The server acquires video in real time through the installed cameras. The video is received as a stream and divided into individual images at a constant frame rate. The server temporarily stores this video data in a database and prepares it for subsequent processing.

[0459] Step 2:

[0460] The server preprocesses the acquired video frames using image processing algorithms. Specifically, filtering is performed to reduce noise, and brightness and contrast are adjusted. This improves the accuracy of pest identification.

[0461] Step 3:

[0462] The server analyzes pre-processed frames using a deep learning model to identify pests. The model is a convolutional neural network (CNN) that has already learned the characteristics of pests. The server labels the identified pests and determines their species and number.

[0463] Step 4:

[0464] The server determines the type and quantity of chemical substances to be produced based on the identification results and transmits this information to the terminal. The instructions from the server include which chemical substances to produce and in what quantities.

[0465] Step 5:

[0466] The terminal generates the appropriate chemical substances based on instructions from the server. If necessary, the terminal mixes the chemical components to create a pheromone in the optimal ratio.

[0467] Step 6:

[0468] The terminal transfers the generated pheromones to a dispersal device and performs dispersal towards the designated area. The dispersal range and duration follow a pre-configured profile.

[0469] Step 7:

[0470] The server comprehensively manages data after the entire process is complete. The server records analysis results and generation / spraying history, which can be used for future analysis and improvement. This further promotes the efficiency of pest management.

[0471] (Example 1)

[0472] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0473] Conventional pest management systems have struggled to effectively control pest damage to crops due to insufficient pest identification accuracy and inefficient chemical application. Furthermore, there has been a lack of integrated system design encompassing identification, generation, application, and management, leaving room for improvement in overall operational efficiency. To address these challenges, more accurate identification functions and efficient chemical generation and application technologies are required.

[0474] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0475] In this invention, the server includes an image analysis means for preprocessing visual data acquired by an image acquisition device and identifying pests, a synthesis means for generating an optimal chemical substance based on the identified type of pest, and a distribution means for spraying the generated chemical substance to a predetermined location. This enables real-time identification of pests, generation of the optimal chemical substance based on the results, and efficient spraying.

[0476] A "video acquisition device" is a device used to collect visual data and is used to acquire images of the environment in real time.

[0477] "Visual data" refers to image information collected by video acquisition devices, and by analyzing this data, it becomes possible to identify pests.

[0478] "Preprocessing" refers to the initial stages of processing that include denoising and adjusting the data to improve the quality of the raw data and facilitate analysis.

[0479] "Pests" refer to insects that are harmful to crops, and the goal is to improve agricultural productivity by controlling their occurrence.

[0480] "Image analysis means" refers to technical means for identifying and classifying objects based on visual data, and it is common to use machine learning algorithms.

[0481] A "convolutional neural network" is a type of deep learning specifically designed for image analysis, enabling it to automatically learn and identify features within images.

[0482] A "synthesis means" is a technical means for compounding chemical substances based on identified targets, and allows for dynamic changes in the mixing ratio.

[0483] "Chemical substances" refer to substances produced for pest control, which, when formulated appropriately, act effectively against specific pests.

[0484] A "distribution mechanism" is a system for delivering and dispersing the generated chemical substances to a designated location, and plays a role in efficiently delivering substances to the target location.

[0485] An "information management system" is a system mechanism for accumulating collected data and using it for analysis and future decision-making.

[0486] One embodiment of this invention is the construction of a comprehensive pest control system. The main components of the system are a server, terminals, and users.

[0487] The server first uses network cameras as video acquisition devices to acquire visual data in real time from monitored areas such as farms. The acquired visual data is preprocessed internally to remove noise. This makes the visual data suitable for analysis.

[0488] The server uses pre-processed visual data and employs convolutional neural network (CNN) technology to identify pests. This allows for the identification of pest types and numbers, providing crucial information for proceeding to the next step.

[0489] The terminal generates chemical substances based on pest information identified by the server. A dynamically adjustable chemical synthesis device is used for this generation. The synthesized chemical substances are prepared in optimal proportions for the target pest and used in the form of pheromones or other substances.

[0490] The terminal also features a dispensing device for spraying the generated chemicals. This dispensing device can be mounted on drones or autonomous vehicles, allowing for precise delivery to designated spraying points and efficient dispensing.

[0491] Users can access identification data and distribution history provided by the server to check the situation in real time and devise countermeasures as needed. Furthermore, this data will be used to design future countermeasures and improve the system.

[0492] For example, if a user manages a cornfield, the server analyzes visual data from network cameras to identify corn borers. The terminal can then generate the appropriate pheromone and spray it on specific areas of the field. This process enables effective pest control without the use of pesticides.

[0493] An example of a prompt would be, "Please propose the optimal method for generating and distributing pheromones as a pest control measure in a cornfield." This would then generate specific countermeasures using a generative AI model.

[0494] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0495] Step 1:

[0496] The server acquires visual data in real time from network cameras. The input is video footage of the farm captured by the cameras. The server divides this video data frame by frame and applies a noise reduction filter to obtain clear image data as output. Specific operations include dimensionality reduction and removal of unnecessary data.

[0497] Step 2:

[0498] The server identifies pests using a convolutional neural network (CNN) based on pre-processed visual data. The input is pre-processed image frames. The server applies the CNN to recognize and label specific pests from each frame, outputting identified pest information. This process involves feature extraction and pattern matching to effectively classify pests.

[0499] Step 3:

[0500] The terminal generates chemical substances based on pest information output from the server. The input is data regarding the type and number of pests. Using a synthesis device employing dynamic mixing ratios, the terminal produces the optimal chemical substance for a specific pest as output. Specific operations include weighing and mixing chemical components.

[0501] Step 4:

[0502] The terminal disperses the generated chemical substance into a designated area. The input is the generated chemical substance. The terminal controls the dispersal device and performs actions to effectively distribute the chemical substance towards the target area. By using automated vehicles or drones, precise and efficient dispersal is achieved as the output.

[0503] Step 5:

[0504] The server stores data related to identification and distribution, preparing it for later analysis. Input consists of data obtained from all processing steps. The server stores the information in a database and provides analysis results as output data, which can be used for damage prediction and future countermeasure planning. Specific operations include data integration and information visualization.

[0505] (Application Example 1)

[0506] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0507] In industrial production facilities, it is crucial to protect products and equipment from pests while maintaining a hygienic environment. However, conventional methods require regular human inspections, which are time-consuming and labor-intensive, and make it difficult to address pest infestations in real time. Therefore, there is a need for a system that can autonomously detect pests and respond immediately.

[0508] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0509] In this invention, the server includes an image processing means for preprocessing image data acquired by a video acquisition device and identifying pests; a generation means for generating an optimal chemical substance based on the identified type of pest; a dispensing means for dispensing the generated chemical substance to a predetermined location; and a display means for monitoring the occurrence of pests within an industrial production facility using an autonomous mobile device and displaying pest occurrence information in real time. This enables real-time monitoring within the facility and rapid pest control.

[0510] A "video acquisition device" is a device used to acquire video data in real time within an industrial production facility.

[0511] "Image processing means" refers to a technique used to preprocess acquired video data and identify pests.

[0512] "Generation method" refers to a technology for generating the optimal chemical substance according to the identified type of pest.

[0513] "Dispersion methods" refer to techniques for effectively dispersing generated chemical substances to specific locations within industrial production facilities.

[0514] A "data management system" is a system for accumulating identification results and distribution history, and for analyzing them.

[0515] An "autonomous mobile device" is an automated device that moves around within an industrial production facility while monitoring for pest outbreaks.

[0516] "Display means" refers to a device or system for visually displaying pest outbreak information in real time.

[0517] The system that realizes this application example is built with the aim of autonomously detecting pests in industrial production facilities and taking rapid and effective countermeasures. The program of this system is mainly based on the Robot Operating System (ROS) and uses software libraries such as OpenCV and TensorFlow.

[0518] The server acquires video data in real time from video acquisition devices installed at the facility. This data is preprocessed using OpenCV, including noise reduction and frame splitting. Subsequently, a convolutional neural network (CNN) implemented with TensorFlow analyzes the video data and identifies pests. Based on the information of the identified pests, the autonomous mobile device utilizes generation methods to produce the optimal chemical substance and moves to the appropriate location.

[0519] The terminal automatically sprays chemicals generated at its destination. A dynamically adjustable mixing ratio is used for the generation of these chemicals, and they are appropriately distributed by the spraying mechanism. The data obtained during these processes is stored on a server and analyzed via SQLite for data management.

[0520] Users can use a display to check the pest infestation status within the facility in real time. Devices such as smart glasses and head-mounted displays can be used for this purpose. A specific prompt for this application example would be: "Identify the objects contained in this frame, and if cockroaches are detected, initiate the corresponding pheromone production and dispersal process."

[0521] This system enables rapid pest control and automated hygiene management in industrial production facilities. It's a solution that supports more efficient operations while maintaining optimal hygiene conditions within the facility at all times.

[0522] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0523] Step 1:

[0524] The server acquires video data in real time from video acquisition devices installed in industrial production facilities. This video data is used as input, and noise reduction and frame splitting are performed using OpenCV. The output is processed, clean image data.

[0525] Step 2:

[0526] The server inputs preprocessed image data into a convolutional neural network (CNN) built with TensorFlow. The CNN extracts features from the image and performs data calculations to identify pests. The output is the type and location information of the identified pests.

[0527] Step 3:

[0528] Based on the pest type obtained in Step 2, the server uses a generation AI model to form prompts for generating the optimal chemical substance. This prompt is sent to the terminal as input. The output is a recipe for a chemical substance suitable for the specific pest.

[0529] Step 4:

[0530] The terminal dynamically adjusts the mixing ratio to produce chemical substances based on the chemical recipe received from the server. Specifically, this production process involves precisely measuring and mixing different chemicals. The resulting chemical substances are then output.

[0531] Step 5:

[0532] The terminal moves the generated chemical substance to a designated location and prepares it for dispersal using an autonomous mobile device. It moves to the specified position and operates the spraying device to effectively disperse the chemical substance. The output is the chemical substance dispersed in the target area.

[0533] Step 6:

[0534] The server stores all data obtained from all processes in an SQLite database and performs analysis as a data management tool. This includes organizing and analyzing identification results and distribution history. An analysis report accessible to the user is then generated.

[0535] Step 7:

[0536] Users can check pest outbreak information in real time using display devices. The analysis information generated in step 6 is displayed on smart glasses or a head-mounted display. The output is a visually displayed dashboard of the pest management status.

[0537] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0538] As an embodiment of the present invention, a specific implementation example in which an emotion engine is incorporated into a pest management system will be described. The emotion engine recognizes the user's emotional state and adjusts the system's operation based on that state.

[0539] System Overview

[0540] This system provides a more flexible and ergonomic agricultural solution by integrating pest control functions with user emotion recognition capabilities. In addition to conventional pest control functions, it adjusts the parameters of chemical generation and spraying according to the user's emotional state, enabling operation that combines safety and efficiency.

[0541] Program processing

[0542] Video acquisition and preprocessing

[0543] The server acquires video footage from the installed cameras. This video data is used for pest detection and as input data for the emotion engine. The server divides the image data frame by frame and performs noise reduction.

[0544] Pest identification

[0545] The server utilizes deep learning models to identify pests from pre-processed frames. This allows for rapid and accurate identification of pest types and distributions, providing foundational data for appropriate responses.

[0546] emotion recognition

[0547] The server analyzes the voice data provided by the user and processes it using an emotion engine. This process determines the user's emotional state, and this data influences subsequent decision-making processes.

[0548] Pheromone production and dispersal regulation

[0549] Based on emotional data obtained from the emotion engine, the device adjusts its pheromone generation and dispersal settings. For example, if the user is feeling stressed, the system increases the amount of pheromones produced to enable a quicker response.

[0550] Data management and proposal of improvement measures.

[0551] The server accumulates data from all processes and provides a follow-up function that suggests more efficient pest control methods through daily analysis. Users can utilize this data to select the optimal agricultural methods tailored to the environment and their emotions.

[0552] Specific example

[0553] When the emotion engine detects that a user is experiencing stress, the terminal will send a command from the server to spray a larger amount of pheromones in areas heavily damaged by corn borers. This enables rapid pest control and allows for agricultural management that is tailored to the user's emotional state.

[0554] This design reduces pest damage to crops, alleviates the mental burden on users, and enables efficient agricultural management.

[0555] The following describes the processing flow.

[0556] Step 1:

[0557] The server acquires video data in real time from network cameras installed on the farm. This video is then divided into frames and pre-processed to remove noise.

[0558] Step 2:

[0559] The server identifies specific pests by analyzing pre-processed frames using a deep learning convolutional neural network (CNN). At this stage, various pests, such as corn borers and spider mites, are classified and detected.

[0560] Step 3:

[0561] The user provides voice input to the system. The server analyzes this voice data using an emotion engine to determine the user's emotional state, such as joy, stress, or anxiety.

[0562] Step 4:

[0563] The server combines the results from the emotion engine and the pest detection results to determine the type and amount of pheromones to be produced. This information is then sent to the terminal as an instruction to produce pheromones.

[0564] Step 5:

[0565] The device prepares the necessary chemicals and generates pheromones based on instructions from the server. In particular, adjustments are made to increase the amount of pheromones produced when the user is experiencing stress.

[0566] Step 6:

[0567] The device transfers the generated pheromones to a dispersal device and sprays them in a specific area of ​​the designated farm. The timing and range of pheromone dispersal are fine-tuned based on the emotional state.

[0568] Step 7:

[0569] The server collects and stores data about the entire process and records the history of each processing step. Based on this data, users can receive suggestions for improving their future farming plans.

[0570] (Example 2)

[0571] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0572] Conventional pest management systems focus solely on pest identification and response, neglecting consideration for the user's emotional state. This can lead to a stressful user experience, hindering efficient and comfortable operation. The present invention aims to enable flexible system operation that takes the user's emotional state into account.

[0573] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0574] In this invention, the server includes an image analysis means for preprocessing visual information obtained by an image acquisition means and identifying pests; a generation means for generating an optimal chemical substance based on the identified type of pest; a spraying means for spraying the generated chemical substance to a predetermined location; an emotion recognition means for analyzing the user's emotions and adjusting system operation; and a data management means for accumulating and analyzing identification results, spraying history, and user emotion information and suggesting improvement measures. This enables efficient and comfortable pest management that responds to the user's emotional state.

[0575] "Image acquisition means" refers to equipment or devices installed to capture visual information, and which have the function of acquiring image data necessary for the system.

[0576] "Image analysis means" refers to a technology that has the function of identifying and analyzing a specific object from acquired visual information, and is used to determine the type and number of pests.

[0577] "Generation means" refers to a device or group of devices that has the function of producing the optimal chemical substance based on the identified target, and which can be adjusted according to dynamic conditions.

[0578] A "dispersion device" is a device that has a mechanism for distributing and spraying the generated chemical substance to a specific location, enabling spraying in a precise amount and over a wide area.

[0579] "Emotion recognition means" refers to a technology that has the function of determining the emotional state from the user's speech and actions, and adjusting the operation of the entire system based on the results.

[0580] A "data management system" is a system that stores identification results, distribution history, and user sentiment information, and analyzes and evaluates this data to suggest more efficient operational methods.

[0581] As an embodiment of this invention, the system is an integrated management device that enables pest management and flexible responses based on the user's emotional state. Specifically, the server uses visual information obtained in real time through video acquisition equipment and image analysis software to remove noise and identify pests. By using a neural network for image analysis, the types and distribution of pests can be accurately grasped.

[0582] The server further receives the user's voice input and analyzes the user's emotional state by utilizing a generative AI model with emotion recognition capabilities. This analysis is used by the device to dynamically adjust the amount of pheromones produced and dispersed. The generated pheromones are then effectively distributed to the necessary locations by appropriate dispersal equipment, minimizing the impact of pests.

[0583] For example, if the emotion engine detects that a user is feeling stressed, the system will automatically configure itself to spray more pheromones in areas where there is a high incidence of pests such as corn borers.

[0584] Furthermore, the server stores all identification results, pheromone dispersal history, and user sentiment information in a database and uses AI technology to perform daily analyses, presenting users with improved and more efficient pest control measures. This allows users to select agricultural methods based on the latest data.

[0585] An example of a prompt sentence for input to a generating AI model is, "Please explain the countermeasures to take when a user experiences stress while using the pest management system."

[0586] This configuration allows the system to achieve fully automated agricultural management that combines pest control and emotional considerations based on real-time data and user sentiment, contributing to improved productivity and user experience on the farm.

[0587] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0588] Step 1:

[0589] The server acquires real-time visual information using video acquisition equipment. This video data is used as input for the next analysis process. By dividing the acquired data frame by frame and performing image preprocessing to remove noise from each frame, a clear image is obtained. This clear image becomes important data for subsequent pest identification.

[0590] Step 2:

[0591] The server applies a deep learning-based neural network model to preprocessed image data as input. This model identifies the presence of pests and determines the type and location of each pest from each frame. The output provides detailed information about the identified pests. This information is used to determine appropriate countermeasures for each type of pest.

[0592] Step 3:

[0593] The user provides voice data via a dedicated voice input device. The server uses this voice data as input and performs emotion analysis of the user using an emotion engine that utilizes a generative AI model. The emotion engine analyzes the features of the voice and outputs the emotional state the user is feeling. This information about the determined emotional state is used to adjust the system's operation.

[0594] Step 4:

[0595] The device adjusts the pheromone generator based on the pest identification results and the user's emotional state. Based on the output from the emotion engine, it optimizes the amount of pheromone produced and the spraying pattern. Especially when the user is stressed, a rapid response is required, so the amount of pheromone produced is increased and the spraying target area is immediately updated. The output after spraying is the result of pheromone production and spraying.

[0596] Step 5:

[0597] The server stores all identification results, spraying history, and user sentiment data in a database. Using this data as input, it performs daily data analysis and proposes efficient pest management methods and optimal system operation based on user sentiment. The final output is an improvement report presented to the user, enabling them to make more informed decisions regarding future actions.

[0598] (Application Example 2)

[0599] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0600] In modern vehicle use, the emotional state of the driver significantly impacts the driving experience. However, conventional systems have difficulty providing flexible vehicle control in response to the driver's emotions, sometimes compromising a comfortable driving experience. Furthermore, the inability to maintain an appropriate in-vehicle environment leads to insufficient stress reduction for the driver. These challenges need to be addressed.

[0601] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0602] In this invention, the server includes an image processing means for preprocessing image data acquired by a video acquisition device and identifying an object to be identified; a generation means for generating an optimal substance based on the type of identified object; a dispersal means for dispersing the generated substance to a predetermined location; and an emotion recognition means for analyzing the user's emotional state and adjusting the parameters of substance generation and dispersal based on that data. This enables flexible and comfortable control of the in-vehicle environment in accordance with the user's emotional state.

[0603] A "video acquisition device" is a device used to capture video data and provides the visual information that the system needs to process.

[0604] "Image processing means" refers to a function that performs a series of operations to analyze acquired image data and identify a specific object.

[0605] A "generating means" is a device or mechanism that dynamically produces the necessary substances or parameters based on the type of object identified.

[0606] "Dispersion means" refers to a mechanism or device that appropriately distributes or releases the generated substance to the target location.

[0607] "Data management means" refers to a function within a system that manages the process for accumulating and analyzing identification results and distribution history.

[0608] "Emotion recognition means" refers to a function that analyzes the user's emotional state and is responsible for the process of adjusting other system parameters based on that information.

[0609] In the system that implements this application example, the server uses video data obtained from a video acquisition device to perform image processing and identify the target object. This utilizes deep learning models such as convolutional neural networks. The server also uses audio data to perform emotion recognition. For emotion recognition, it uses audio analysis technology to determine the user's emotional state and adjusts the properties and dispersion parameters of the generated substance based on the analysis results.

[0610] As a concrete example, the server uses cameras and microphones inside the vehicle to detect the emotional state of the user from their facial expressions and voice data while they are riding. If the emotion recognition system determines that the user is experiencing stress, the system adjusts the in-vehicle environment. For example, it may change the air conditioning temperature or play relaxing music to improve the user's comfort.

[0611] In this process, the generative AI model used is input with the following prompt:

[0612] "If you are currently feeling stressed, adjust the air conditioning to a comfortable temperature and play a relaxing playlist to create a more comfortable environment in your car."

[0613] This configuration enables flexible and comfortable control of the in-car environment in response to the user's emotional state.

[0614] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0615] Step 1:

[0616] The server acquires video and audio data from cameras and microphones installed inside the vehicle. The input is real-time video and audio, which are used directly in the next processing step.

[0617] Step 2:

[0618] The server divides the acquired video data into frames and performs noise reduction. Then, it uses a deep learning model (particularly a convolutional neural network) to perform face recognition and facial expression analysis to identify the user's emotions. The output of this step is emotion data estimated from the user's facial expressions.

[0619] Step 3:

[0620] The server analyzes the voice data using an emotion recognition engine to determine the user's emotions from the tone and pitch of the voice. The input is voice data, and the output is the analyzed emotional state.

[0621] Step 4:

[0622] The server integrates the emotion data obtained as outputs from steps 2 and 3 to determine the user's overall emotional state. Here, a weighted average of the information obtained from the video and audio is used to derive the user's emotion score. The output of this step is the user's overall emotional evaluation.

[0623] Step 5:

[0624] Based on the results of the emotion assessment, the server generates prompt messages to adjust the in-car environment using a generative AI model. For example, if "stress" is detected, instructions to lower the air conditioning temperature or play relaxing music will be generated. These prompt messages are generated automatically and executed in the following steps.

[0625] Step 6:

[0626] The terminal controls the in-car environment based on prompt messages received from the server. It performs actions such as adjusting the air conditioning and playing specified music through the speakers. The input is the prompt message, and the output is the adjusted in-car environment. This allows the user to comfortably enjoy an in-car space optimized for their emotional state.

[0627] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0628] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0629] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0630] [Fourth Embodiment]

[0631] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0632] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0633] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0634] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0635] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0636] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0637] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0638] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0639] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0640] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0641] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0642] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0643] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] As an embodiment for carrying out the present invention, a specific implementation example of a pest control system will be described.

[0645] System Overview

[0646] This system consists of multiple functional modules, primarily performing functions such as video acquisition, image processing, chemical generation, spraying, and data management. The system comprises servers, terminals, and users, each fulfilling its respective role to effectively suppress pest damage.

[0647] Program processing

[0648] Video acquisition and preprocessing

[0649] The server acquires real-time video data from network cameras installed on the farm. This provides a foundation for continuously monitoring pest outbreaks. To efficiently process the video data, the server divides it into frames and pre-processes it to reduce noise as needed.

[0650] Pest identification

[0651] The server uses pre-processed frames to apply a convolutional neural network (CNN), an AI model, to identify specific pests. In this process, each identified pest is assigned a specific label, and processing proceeds based on that label.

[0652] Pheromone production and dispersal

[0653] The terminal generates the optimal pheromone based on pest information identified by the server. By dynamically adjusting the mixing ratio of chemicals, it is possible to generate the most effective pheromone for specific pests. The generated pheromone is automatically moved by the terminal to the application location and appropriately dispersed using a spraying device.

[0654] Data management and analysis

[0655] The server stores data obtained from each process, preparing it for later analysis. This data includes the types and numbers of identified pests, and the amount of pheromones sprayed. Users can access this data to help predict damage and develop further countermeasures.

[0656] Specific example

[0657] When a user manages a cornfield, the server analyzes the video feed from the camera and identifies the corn borer. Instructions for generating the necessary pheromones are sent to the terminal, which then generates the pheromone best suited to the corn borer and sprays it in specific areas of the field. Because this process is optimized for each pest species, effective pest control is achieved without the use of pesticides.

[0658] This system is a useful solution that can significantly reduce crop damage caused by pests while also improving the efficiency of agricultural work.

[0659] The following describes the processing flow.

[0660] Step 1:

[0661] The server acquires video in real time through the installed cameras. The video is received as a stream and divided into individual images at a constant frame rate. The server temporarily stores this video data in a database and prepares it for subsequent processing.

[0662] Step 2:

[0663] The server preprocesses the acquired video frames using image processing algorithms. Specifically, filtering is performed to reduce noise, and brightness and contrast are adjusted. This improves the accuracy of pest identification.

[0664] Step 3:

[0665] The server analyzes pre-processed frames using a deep learning model to identify pests. The model is a convolutional neural network (CNN) that has already learned the characteristics of pests. The server labels the identified pests and determines their species and number.

[0666] Step 4:

[0667] The server determines the type and quantity of chemical substances to be produced based on the identification results and transmits this information to the terminal. The instructions from the server include which chemical substances to produce and in what quantities.

[0668] Step 5:

[0669] The terminal generates the appropriate chemical substances based on instructions from the server. If necessary, the terminal mixes the chemical components to create a pheromone in the optimal ratio.

[0670] Step 6:

[0671] The terminal transfers the generated pheromones to a dispersal device and performs dispersal towards the designated area. The dispersal range and duration follow a pre-configured profile.

[0672] Step 7:

[0673] The server comprehensively manages data after the entire process is complete. The server records analysis results and generation / spraying history, which can be used for future analysis and improvement. This further promotes the efficiency of pest management.

[0674] (Example 1)

[0675] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] Conventional pest management systems have struggled to effectively control pest damage to crops due to insufficient pest identification accuracy and inefficient chemical application. Furthermore, there has been a lack of integrated system design encompassing identification, generation, application, and management, leaving room for improvement in overall operational efficiency. To address these challenges, more accurate identification functions and efficient chemical generation and application technologies are required.

[0677] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0678] In this invention, the server includes an image analysis means for preprocessing visual data acquired by an image acquisition device and identifying pests, a synthesis means for generating an optimal chemical substance based on the identified type of pest, and a distribution means for spraying the generated chemical substance to a predetermined location. This enables real-time identification of pests, generation of the optimal chemical substance based on the results, and efficient spraying.

[0679] A "video acquisition device" is a device used to collect visual data and is used to acquire images of the environment in real time.

[0680] "Visual data" refers to image information collected by video acquisition devices, and by analyzing this data, it becomes possible to identify pests.

[0681] "Preprocessing" refers to the initial stages of processing that include denoising and adjusting the data to improve the quality of the raw data and facilitate analysis.

[0682] "Pests" refer to insects that are harmful to crops, and the goal is to improve agricultural productivity by controlling their occurrence.

[0683] "Image analysis means" refers to technical means for identifying and classifying objects based on visual data, and it is common to use machine learning algorithms.

[0684] A "convolutional neural network" is a type of deep learning specifically designed for image analysis, enabling it to automatically learn and identify features within images.

[0685] A "synthesis means" is a technical means for compounding chemical substances based on identified targets, and allows for dynamic changes in the mixing ratio.

[0686] "Chemical substances" refer to substances produced for pest control, which, when formulated appropriately, act effectively against specific pests.

[0687] A "distribution mechanism" is a system for delivering and dispersing the generated chemical substances to a designated location, and plays a role in efficiently delivering substances to the target location.

[0688] An "information management system" is a system mechanism for accumulating collected data and using it for analysis and future decision-making.

[0689] One embodiment of this invention is the construction of a comprehensive pest control system. The main components of the system are a server, terminals, and users.

[0690] The server first uses network cameras as video acquisition devices to acquire visual data in real time from monitored areas such as farms. The acquired visual data is preprocessed internally to remove noise. This makes the visual data suitable for analysis.

[0691] The server uses pre-processed visual data and employs convolutional neural network (CNN) technology to identify pests. This allows for the identification of pest types and numbers, providing crucial information for proceeding to the next step.

[0692] The terminal generates chemical substances based on pest information identified by the server. A dynamically adjustable chemical synthesis device is used for this generation. The synthesized chemical substances are prepared in optimal proportions for the target pest and used in the form of pheromones or other substances.

[0693] The terminal also features a dispensing device for spraying the generated chemicals. This dispensing device can be mounted on drones or autonomous vehicles, allowing for precise delivery to designated spraying points and efficient dispensing.

[0694] Users can access identification data and distribution history provided by the server to check the situation in real time and devise countermeasures as needed. Furthermore, this data will be used to design future countermeasures and improve the system.

[0695] For example, if a user manages a cornfield, the server analyzes visual data from network cameras to identify corn borers. The terminal can then generate the appropriate pheromone and spray it on specific areas of the field. This process enables effective pest control without the use of pesticides.

[0696] An example of a prompt would be, "Please propose the optimal method for generating and distributing pheromones as a pest control measure in a cornfield." This would then generate specific countermeasures using a generative AI model.

[0697] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0698] Step 1:

[0699] The server acquires visual data in real time from network cameras. The input is video footage of the farm captured by the cameras. The server divides this video data frame by frame and applies a noise reduction filter to obtain clear image data as output. Specific operations include dimensionality reduction and removal of unnecessary data.

[0700] Step 2:

[0701] The server identifies pests using a convolutional neural network (CNN) based on pre-processed visual data. The input is pre-processed image frames. The server applies the CNN to recognize and label specific pests from each frame, outputting identified pest information. This process involves feature extraction and pattern matching to effectively classify pests.

[0702] Step 3:

[0703] The terminal generates chemical substances based on pest information output from the server. The input is data regarding the type and number of pests. Using a synthesis device employing dynamic mixing ratios, the terminal produces the optimal chemical substance for a specific pest as output. Specific operations include weighing and mixing chemical components.

[0704] Step 4:

[0705] The terminal disperses the generated chemical substance into a designated area. The input is the generated chemical substance. The terminal controls the dispersal device and performs actions to effectively distribute the chemical substance towards the target area. By using automated vehicles or drones, precise and efficient dispersal is achieved as the output.

[0706] Step 5:

[0707] The server stores data related to identification and distribution, preparing it for later analysis. Input consists of data obtained from all processing steps. The server stores the information in a database and provides analysis results as output data, which can be used for damage prediction and future countermeasure planning. Specific operations include data integration and information visualization.

[0708] (Application Example 1)

[0709] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0710] In industrial production facilities, it is crucial to protect products and equipment from pests while maintaining a hygienic environment. However, conventional methods require regular human inspections, which are time-consuming and labor-intensive, and make it difficult to address pest infestations in real time. Therefore, there is a need for a system that can autonomously detect pests and respond immediately.

[0711] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0712] In this invention, the server includes an image processing means for preprocessing image data acquired by a video acquisition device and identifying pests; a generation means for generating an optimal chemical substance based on the identified type of pest; a dispensing means for dispensing the generated chemical substance to a predetermined location; and a display means for monitoring the occurrence of pests within an industrial production facility using an autonomous mobile device and displaying pest occurrence information in real time. This enables real-time monitoring within the facility and rapid pest control.

[0713] A "video acquisition device" is a device used to acquire video data in real time within an industrial production facility.

[0714] "Image processing means" refers to a technique used to preprocess acquired video data and identify pests.

[0715] "Generation method" refers to a technology for generating the optimal chemical substance according to the identified type of pest.

[0716] "Dispersion methods" refer to techniques for effectively dispersing generated chemical substances to specific locations within industrial production facilities.

[0717] A "data management system" is a system for accumulating identification results and distribution history, and for analyzing them.

[0718] An "autonomous mobile device" is an automated device that moves around within an industrial production facility while monitoring for pest outbreaks.

[0719] "Display means" refers to a device or system for visually displaying pest outbreak information in real time.

[0720] The system that realizes this application example is built with the aim of autonomously detecting pests in industrial production facilities and taking rapid and effective countermeasures. The program of this system is mainly based on the Robot Operating System (ROS) and uses software libraries such as OpenCV and TensorFlow.

[0721] The server acquires video data in real time from video acquisition devices installed at the facility. This data is preprocessed using OpenCV, including noise reduction and frame splitting. Subsequently, a convolutional neural network (CNN) implemented with TensorFlow analyzes the video data and identifies pests. Based on the information of the identified pests, the autonomous mobile device utilizes generation methods to produce the optimal chemical substance and moves to the appropriate location.

[0722] The terminal automatically sprays chemicals generated at its destination. A dynamically adjustable mixing ratio is used for the generation of these chemicals, and they are appropriately distributed by the spraying mechanism. The data obtained during these processes is stored on a server and analyzed via SQLite for data management.

[0723] Users can use a display to check the pest infestation status within the facility in real time. Devices such as smart glasses and head-mounted displays can be used for this purpose. A specific prompt for this application example would be: "Identify the objects contained in this frame, and if cockroaches are detected, initiate the corresponding pheromone production and dispersal process."

[0724] This system enables rapid pest control and automated hygiene management in industrial production facilities. It's a solution that supports more efficient operations while maintaining optimal hygiene conditions within the facility at all times.

[0725] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0726] Step 1:

[0727] The server acquires video data in real time from video acquisition devices installed in industrial production facilities. This video data is used as input, and noise reduction and frame splitting are performed using OpenCV. The output is processed, clean image data.

[0728] Step 2:

[0729] The server inputs preprocessed image data into a convolutional neural network (CNN) built with TensorFlow. The CNN extracts features from the image and performs data calculations to identify pests. The output is the type and location information of the identified pests.

[0730] Step 3:

[0731] Based on the pest type obtained in Step 2, the server uses a generation AI model to form prompts for generating the optimal chemical substance. This prompt is sent to the terminal as input. The output is a recipe for a chemical substance suitable for the specific pest.

[0732] Step 4:

[0733] The terminal dynamically adjusts the mixing ratio to produce chemical substances based on the chemical recipe received from the server. Specifically, this production process involves precisely measuring and mixing different chemicals. The resulting chemical substances are then output.

[0734] Step 5:

[0735] The terminal moves the generated chemical substance to a designated location and prepares it for dispersal using an autonomous mobile device. It moves to the specified position and operates the spraying device to effectively disperse the chemical substance. The output is the chemical substance dispersed in the target area.

[0736] Step 6:

[0737] The server stores all data obtained from all processes in an SQLite database and performs analysis as a data management tool. This includes organizing and analyzing identification results and distribution history. An analysis report accessible to the user is then generated.

[0738] Step 7:

[0739] Users can check pest outbreak information in real time using display devices. The analysis information generated in step 6 is displayed on smart glasses or a head-mounted display. The output is a visually displayed dashboard of the pest management status.

[0740] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0741] As an embodiment of the present invention, a specific implementation example in which an emotion engine is incorporated into a pest management system will be described. The emotion engine recognizes the user's emotional state and adjusts the system's operation based on that state.

[0742] System Overview

[0743] This system provides a more flexible and ergonomic agricultural solution by integrating pest control functions with user emotion recognition capabilities. In addition to conventional pest control functions, it adjusts the parameters of chemical generation and spraying according to the user's emotional state, enabling operation that combines safety and efficiency.

[0744] Program processing

[0745] Video acquisition and preprocessing

[0746] The server acquires video footage from the installed cameras. This video data is used for pest detection and as input data for the emotion engine. The server divides the image data frame by frame and performs noise reduction.

[0747] Pest identification

[0748] The server utilizes deep learning models to identify pests from pre-processed frames. This allows for rapid and accurate identification of pest types and distributions, providing foundational data for appropriate responses.

[0749] emotion recognition

[0750] The server analyzes the voice data provided by the user and processes it using an emotion engine. This process determines the user's emotional state, and this data influences subsequent decision-making processes.

[0751] Pheromone production and dispersal regulation

[0752] Based on emotional data obtained from the emotion engine, the device adjusts its pheromone generation and dispersal settings. For example, if the user is feeling stressed, the system increases the amount of pheromones produced to enable a quicker response.

[0753] Data management and proposal of improvement measures.

[0754] The server accumulates data from all processes and provides a follow-up function that suggests more efficient pest control methods through daily analysis. Users can utilize this data to select the optimal agricultural methods tailored to the environment and their emotions.

[0755] Specific example

[0756] When the emotion engine detects that a user is experiencing stress, the terminal will send a command from the server to spray a larger amount of pheromones in areas heavily damaged by corn borers. This enables rapid pest control and allows for agricultural management that is tailored to the user's emotional state.

[0757] This design reduces pest damage to crops, alleviates the mental burden on users, and enables efficient agricultural management.

[0758] The following describes the processing flow.

[0759] Step 1:

[0760] The server acquires video data in real time from network cameras installed on the farm. This video is then divided into frames and pre-processed to remove noise.

[0761] Step 2:

[0762] The server identifies specific pests by analyzing pre-processed frames using a deep learning convolutional neural network (CNN). At this stage, various pests, such as corn borers and spider mites, are classified and detected.

[0763] Step 3:

[0764] The user provides voice input to the system. The server analyzes this voice data using an emotion engine to determine the user's emotional state, such as joy, stress, or anxiety.

[0765] Step 4:

[0766] The server combines the results from the emotion engine and the pest detection results to determine the type and amount of pheromones to be produced. This information is then sent to the terminal as an instruction to produce pheromones.

[0767] Step 5:

[0768] The device prepares the necessary chemicals and generates pheromones based on instructions from the server. In particular, adjustments are made to increase the amount of pheromones produced when the user is experiencing stress.

[0769] Step 6:

[0770] The device transfers the generated pheromones to a dispersal device and sprays them in a specific area of ​​the designated farm. The timing and range of pheromone dispersal are fine-tuned based on the emotional state.

[0771] Step 7:

[0772] The server collects and stores data about the entire process and records the history of each processing step. Based on this data, users can receive suggestions for improving their future farming plans.

[0773] (Example 2)

[0774] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0775] Conventional pest management systems focus solely on pest identification and response, neglecting consideration for the user's emotional state. This can lead to a stressful user experience, hindering efficient and comfortable operation. The present invention aims to enable flexible system operation that takes the user's emotional state into account.

[0776] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0777] In this invention, the server includes an image analysis means for preprocessing visual information obtained by an image acquisition means and identifying pests; a generation means for generating an optimal chemical substance based on the identified type of pest; a spraying means for spraying the generated chemical substance to a predetermined location; an emotion recognition means for analyzing the user's emotions and adjusting system operation; and a data management means for accumulating and analyzing identification results, spraying history, and user emotion information and suggesting improvement measures. This enables efficient and comfortable pest management that responds to the user's emotional state.

[0778] "Image acquisition means" refers to equipment or devices installed to capture visual information, and which have the function of acquiring image data necessary for the system.

[0779] "Image analysis means" refers to a technology that has the function of identifying and analyzing a specific object from acquired visual information, and is used to determine the type and number of pests.

[0780] "Generation means" refers to a device or group of devices that has the function of producing the optimal chemical substance based on the identified target, and which can be adjusted according to dynamic conditions.

[0781] A "dispersion device" is a device that has a mechanism for distributing and spraying the generated chemical substance to a specific location, enabling spraying in a precise amount and over a wide area.

[0782] "Emotion recognition means" refers to a technology that has the function of determining the emotional state from the user's speech and actions, and adjusting the operation of the entire system based on the results.

[0783] A "data management system" is a system that stores identification results, distribution history, and user sentiment information, and analyzes and evaluates this data to suggest more efficient operational methods.

[0784] As an embodiment of this invention, the system is an integrated management device that enables pest management and flexible responses based on the user's emotional state. Specifically, the server uses visual information obtained in real time through video acquisition equipment and image analysis software to remove noise and identify pests. By using a neural network for image analysis, the types and distribution of pests can be accurately grasped.

[0785] The server further receives the user's voice input and analyzes the user's emotional state by utilizing a generative AI model with emotion recognition capabilities. This analysis is used by the device to dynamically adjust the amount of pheromones produced and dispersed. The generated pheromones are then effectively distributed to the necessary locations by appropriate dispersal equipment, minimizing the impact of pests.

[0786] For example, if the emotion engine detects that a user is feeling stressed, the system will automatically configure itself to spray more pheromones in areas where there is a high incidence of pests such as corn borers.

[0787] Furthermore, the server stores all identification results, pheromone dispersal history, and user sentiment information in a database and uses AI technology to perform daily analyses, presenting users with improved and more efficient pest control measures. This allows users to select agricultural methods based on the latest data.

[0788] An example of a prompt sentence for input to a generating AI model is, "Please explain the countermeasures to take when a user experiences stress while using the pest management system."

[0789] This configuration allows the system to achieve fully automated agricultural management that combines pest control and emotional considerations based on real-time data and user sentiment, contributing to improved productivity and user experience on the farm.

[0790] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0791] Step 1:

[0792] The server acquires real-time visual information using video acquisition equipment. This video data is used as input for the next analysis process. By dividing the acquired data frame by frame and performing image preprocessing to remove noise from each frame, a clear image is obtained. This clear image becomes important data for subsequent pest identification.

[0793] Step 2:

[0794] The server applies a deep learning-based neural network model to preprocessed image data as input. This model identifies the presence of pests and determines the type and location of each pest from each frame. The output provides detailed information about the identified pests. This information is used to determine appropriate countermeasures for each type of pest.

[0795] Step 3:

[0796] The user provides voice data via a dedicated voice input device. The server uses this voice data as input and performs emotion analysis of the user using an emotion engine that utilizes a generative AI model. The emotion engine analyzes the features of the voice and outputs the emotional state the user is feeling. This information about the determined emotional state is used to adjust the system's operation.

[0797] Step 4:

[0798] The device adjusts the pheromone generator based on the pest identification results and the user's emotional state. Based on the output from the emotion engine, it optimizes the amount of pheromone produced and the spraying pattern. Especially when the user is stressed, a rapid response is required, so the amount of pheromone produced is increased and the spraying target area is immediately updated. The output after spraying is the result of pheromone production and spraying.

[0799] Step 5:

[0800] The server stores all identification results, spraying history, and user sentiment data in a database. Using this data as input, it performs daily data analysis and proposes efficient pest management methods and optimal system operation based on user sentiment. The final output is an improvement report presented to the user, enabling them to make more informed decisions regarding future actions.

[0801] (Application Example 2)

[0802] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0803] In modern vehicle use, the emotional state of the driver significantly impacts the driving experience. However, conventional systems have difficulty providing flexible vehicle control in response to the driver's emotions, sometimes compromising a comfortable driving experience. Furthermore, the inability to maintain an appropriate in-vehicle environment leads to insufficient stress reduction for the driver. These challenges need to be addressed.

[0804] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0805] In this invention, the server includes an image processing means for preprocessing image data acquired by a video acquisition device and identifying an object to be identified; a generation means for generating an optimal substance based on the type of identified object; a dispersal means for dispersing the generated substance to a predetermined location; and an emotion recognition means for analyzing the user's emotional state and adjusting the parameters of substance generation and dispersal based on that data. This enables flexible and comfortable control of the in-vehicle environment in accordance with the user's emotional state.

[0806] A "video acquisition device" is a device used to capture video data and provides the visual information that the system needs to process.

[0807] "Image processing means" refers to a function that performs a series of operations to analyze acquired image data and identify a specific object.

[0808] A "generating means" is a device or mechanism that dynamically produces the necessary substances or parameters based on the type of object identified.

[0809] "Dispersion means" refers to a mechanism or device that appropriately distributes or releases the generated substance to the target location.

[0810] "Data management means" refers to a function within a system that manages the process for accumulating and analyzing identification results and distribution history.

[0811] "Emotion recognition means" refers to a function that analyzes the user's emotional state and is responsible for the process of adjusting other system parameters based on that information.

[0812] In the system that implements this application example, the server uses video data obtained from a video acquisition device to perform image processing and identify the target object. This utilizes deep learning models such as convolutional neural networks. The server also uses audio data to perform emotion recognition. For emotion recognition, it uses audio analysis technology to determine the user's emotional state and adjusts the properties and dispersion parameters of the generated substance based on the analysis results.

[0813] As a concrete example, the server uses cameras and microphones inside the vehicle to detect the emotional state of the user from their facial expressions and voice data while they are riding. If the emotion recognition system determines that the user is experiencing stress, the system adjusts the in-vehicle environment. For example, it may change the air conditioning temperature or play relaxing music to improve the user's comfort.

[0814] In this process, the generative AI model used is input with the following prompt:

[0815] "If you are currently feeling stressed, adjust the air conditioning to a comfortable temperature and play a relaxing playlist to create a more comfortable environment in your car."

[0816] This configuration enables flexible and comfortable control of the in-car environment in response to the user's emotional state.

[0817] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0818] Step 1:

[0819] The server acquires video and audio data from cameras and microphones installed inside the vehicle. The input is real-time video and audio, which are used directly in the next processing step.

[0820] Step 2:

[0821] The server divides the acquired video data into frames and performs noise reduction. Then, it uses a deep learning model (particularly a convolutional neural network) to perform face recognition and facial expression analysis to identify the user's emotions. The output of this step is emotion data estimated from the user's facial expressions.

[0822] Step 3:

[0823] The server analyzes the voice data using an emotion recognition engine to determine the user's emotions from the tone and pitch of the voice. The input is voice data, and the output is the analyzed emotional state.

[0824] Step 4:

[0825] The server integrates the emotion data obtained as outputs from steps 2 and 3 to determine the user's overall emotional state. Here, a weighted average of the information obtained from the video and audio is used to derive the user's emotion score. The output of this step is the user's overall emotional evaluation.

[0826] Step 5:

[0827] Based on the results of the emotion assessment, the server generates prompt messages to adjust the in-car environment using a generative AI model. For example, if "stress" is detected, instructions to lower the air conditioning temperature or play relaxing music will be generated. These prompt messages are generated automatically and executed in the following steps.

[0828] Step 6:

[0829] The terminal controls the in-car environment based on prompt messages received from the server. It performs actions such as adjusting the air conditioning and playing specified music through the speakers. The input is the prompt message, and the output is the adjusted in-car environment. This allows the user to comfortably enjoy an in-car space optimized for their emotional state.

[0830] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0831] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0832] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0833] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0834] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0835] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0836] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0837] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0838] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0839] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0840] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0841] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0842] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0843] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0844] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0845] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0846] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0847] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0848] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0849] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0850] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0851] The following is further disclosed regarding the embodiments described above.

[0852] (Claim 1)

[0853] Image processing means for preprocessing image data acquired by a video acquisition device to identify pests,

[0854] A means for producing an optimal chemical substance based on the identified type of pest,

[0855] A means for dispersing the generated chemical substance to a designated location,

[0856] A data management means for accumulating and analyzing identification results and dispersal history,

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, wherein the image processing means uses a convolutional neural network to identify pests.

[0860] (Claim 3)

[0861] The system according to claim 1, wherein the generating means dynamically adjusts the mixing ratio of chemical substances to generate a chemical substance suitable for pests.

[0862] "Example 1"

[0863] (Claim 1)

[0864] Image analysis means for preprocessing visual data acquired by an image acquisition device and identifying pests,

[0865] A synthesis method for producing the optimal chemical substance based on the identified type of pest,

[0866] A distribution means for spraying the generated chemical substance to a designated location,

[0867] Information management means for accumulating and analyzing identification results and dispersal history,

[0868] An information processing system that includes this.

[0869] (Claim 2)

[0870] The information processing system according to claim 1, wherein the image analysis means uses a convolutional neural network to identify pests.

[0871] (Claim 3)

[0872] The information processing system according to claim 1, wherein the synthesis means dynamically adjusts the mixing ratio of chemical substances to produce a chemical substance suitable for pests.

[0873] "Application Example 1"

[0874] (Claim 1)

[0875] Image processing means for preprocessing image data acquired by a video acquisition device to identify pests,

[0876] A means for producing an optimal chemical substance based on the identified type of pest,

[0877] A means for dispersing the generated chemical substance to a designated location,

[0878] A data management means for accumulating and analyzing identification results and dispersal history,

[0879] Within industrial production facilities, autonomous mobile devices are used to monitor for pest outbreaks.

[0880] A display device that shows pest outbreak information in real time,

[0881] A system that includes this.

[0882] (Claim 2)

[0883] The system according to claim 1, wherein the image processing means uses a convolutional neural network to identify pests.

[0884] (Claim 3)

[0885] The system according to claim 1, wherein the generating means dynamically adjusts the mixing ratio of chemical substances to generate a chemical substance suitable for pests and sprays it in an optimal arrangement at an industrial production facility.

[0886] "Example 2 of combining an emotion engine"

[0887] (Claim 1)

[0888] Image analysis means for preprocessing visual information obtained by image acquisition means and identifying pests,

[0889] A means for producing an optimal chemical substance based on the identified type of pest,

[0890] A means for dispersing the generated chemical substance to a designated location,

[0891] An emotion recognition means that analyzes user emotions and adjusts system operation,

[0892] A data management system that stores and analyzes identification results, distribution history, and user sentiment information to suggest improvement measures,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, wherein the image analysis means uses a neural network to identify pests.

[0896] (Claim 3)

[0897] The system according to claim 1, wherein the generating means generates a chemical substance suitable for pests based on emotion recognition means that dynamically adjusts the amount of chemical substance generated and the amount of spraying.

[0898] "Application example 2 of combining emotional engines"

[0899] (Claim 1)

[0900] Image processing means for preprocessing image data acquired by a video acquisition device and identifying the object to be identified,

[0901] A means for producing an optimal substance based on the type of identified target,

[0902] A means for dispersing the generated substance to a predetermined location,

[0903] A data management means for accumulating and analyzing identification results and dispersal history,

[0904] An emotion recognition means that analyzes the emotional state of the user and adjusts the parameters of substance generation and dispersal based on that data,

[0905] A system that includes this.

[0906] (Claim 2)

[0907] The system according to claim 1, wherein the image processing means identifies an object using a convolutional neural network.

[0908] (Claim 3)

[0909] The system according to claim 1, wherein the generating means dynamically adjusts the mixing ratio of substances to generate a substance suitable for the target, and optimizes the system by reflecting the analysis results of the emotion recognition means, taking into account the user's emotions. [Explanation of Symbols]

[0910] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Image processing means for preprocessing image data acquired by a video acquisition device to identify pests, A means for producing an optimal chemical substance based on the identified type of pest, A means for dispersing the generated chemical substance to a designated location, A data management means for accumulating and analyzing identification results and dispersal history, A system that includes this.

2. The system according to claim 1, wherein the image processing means uses a convolutional neural network to identify pests.

3. The system according to claim 1, wherein the generating means dynamically adjusts the mixing ratio of chemical substances to generate a chemical substance suitable for pests.

Images