system

Abstract

We provide the system. [Solution] A means for analyzing images of plants taken and executing an algorithm to identify the characteristics and condition of those plants, A means for generating advice on how to cultivate plants based on identified characteristics and conditions, A means of providing recommended materials based on the generated advice and supporting the purchase of those materials, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When growing plants, there are problems that beginners and people with little experience cannot quickly and easily make correct diagnoses and take appropriate measures regarding the deterioration of the health of plants and the inappropriate growing environment they face. Furthermore, it is difficult to select the optimal growing method and materials for each plant, so there is also a problem that plants cannot be grown efficiently.

Means for Solving the Problems

[0005] To solve this problem, the present invention provides means for analyzing images of photographed plants and identifying the characteristics and condition of those plants. It also provides means for generating advice on how to cultivate the plants based on the identified characteristics and condition. Furthermore, by employing a system that includes means for suggesting recommended materials in accordance with the generated advice and assisting in the purchase of those materials, the present invention solves the cultivation challenges faced by users.

[0006] "Images of photographed plants" refer to digital data that visually records the condition of plants, acquired by the user using a device.

[0007] An "algorithm" is a set of computational steps designed to achieve a specific purpose, and this one is used to analyze images of plants to identify their features and condition.

[0008] "Characteristics and condition" refers to factors that contribute to the identification and evaluation of plants, such as plant species, health status, nutritional status, and environmental adaptability.

[0009] "Advice" refers to information that suggests appropriate actions to improve the health and growing environment of plants.

[0010] "Recommended materials" refer to fertilizers and other supplementary products necessary to promote plant growth and improve their health.

[0011] "To present" means to provide users with information or options, encouraging them to understand and make decisions.

[0012] "Supporting purchases" means facilitating the process of users acquiring recommended materials and providing the necessary information and means for purchasing. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a processor with a reference number (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a RAM (Random Access Memory) with a reference number is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a storage with a reference number is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, a communication I / F (Interface) with a reference number is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that utilizes devices such as smartphones and tablets owned by plant growers to easily monitor the condition of plants and to support the selection of appropriate cultivation methods and materials.

[0035] The first step for the user is to take a picture of their plant using a smartphone or other device. The device sends this picture to a server in the cloud. The server then runs a dedicated algorithm based on the received image. This algorithm is pre-trained using machine learning techniques and can identify the type of plant, as well as the color, shape, and condition of its leaves and stems, through image analysis.

[0036] Based on the analysis results, the server identifies potential problems with the plant. For example, yellowing leaves may indicate a nitrogen deficiency, and unusual leaf shapes may suggest disease. Based on these analysis results, the server generates specific advice, including which nutrients are lacking and whether adjustments to sunlight or watering should be made.

[0037] Furthermore, the server provides information about the materials needed to solve the problem. This information includes details about the best materials for the plant type and the problem it is facing, such as specific fertilizers and soil amendments. The terminal notifies the user of this information and helps the user purchase the materials directly online.

[0038] As a concrete example, consider a scenario where a user takes a picture of their rose bush at home and sends it to the system. The server diagnoses that the rose is infected with powdery mildew and generates advice to use a fungicide effective against this disease. It also provides product information and online purchase links for the appropriate fungicide. This entire process allows the user to address plant problems quickly and effectively.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] Users take photos of plants using their smartphones or tablets. When taking photos, users pay attention to the angle and lighting so that the entire plant can be clearly seen.

[0042] Step 2:

[0043] The device receives the photographed plant image and adjusts the image size and resolution as needed. Next, the device sends the adjusted image data to a server in the cloud.

[0044] Step 3:

[0045] The server receives the image sent from the terminal. The server passes this image to an image analysis module, which uses a deep learning algorithm to analyze the characteristics and condition of the plants in the image. This analysis identifies the type of plant, leaf color and lesions, shape, and other characteristics.

[0046] Step 4:

[0047] The server evaluates the plant's health based on the analysis results. The evaluation process lists known problems (e.g., nutrient deficiencies, pest and disease effects) and determines whether those problems exist.

[0048] Step 5:

[0049] The server generates advice for the user based on the evaluation results. This advice includes things like watering, adjusting sunlight, and treating diseases.

[0050] Step 6:

[0051] The server lists recommended materials based on the plant type and condition. The list includes the specific name of the material, its effects, and a link to purchase it.

[0052] Step 7:

[0053] The device receives advice and recommended materials information sent from the server and notifies the user. The user can review the displayed information and purchase the recommended materials via the online store if necessary.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] One of the challenges faced by plant growers is the difficulty in accurately assessing plant health and quickly and effectively finding cultivation methods and problem-solving solutions. In particular, making expert judgments based on plant species and conditions, and providing cultivation advice and selecting materials accordingly, requires advanced knowledge and experience. Therefore, there is a need for methods that allow ordinary users to easily obtain this information and manage plant health.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes means for analyzing images of captured plants and executing an algorithm using machine learning techniques to identify the type of plant and the color, shape, and condition of its leaves and stems; means for automatically generating specific advice on how to cultivate and manage the plant based on the identified type, characteristics, and condition; and means including an interface that provides recommended material information based on the generated advice and assists in the online purchase of such materials. This makes it possible for users to easily understand the health of plants and quickly perform appropriate cultivation and problem solving without requiring specialized knowledge.

[0059] "Analyzing plant images" refers to processing plant image data transmitted from a terminal on a computer to identify the type of plant, as well as the color, shape, and condition of its leaves and stems.

[0060] An "algorithm using machine learning technology" is a technique for training computers to make accurate inferences based on vast amounts of data. In the context of plant image analysis, it refers to a program used to classify and recognize the features of image data based on a learned model.

[0061] "Specific advice on plant cultivation and management methods" refers to instructions and suggestions provided according to the analyzed state of the plant, and includes information on nutritional supplementation and environmental adjustments to maintain and improve plant health.

[0062] "Recommended materials information" refers to detailed information about products such as fertilizers, soil amendments, and fungicides that are recommended for use to address specific plant conditions or problems.

[0063] An "online purchase support interface" refers to a user interface that allows users to easily purchase recommended materials, and includes elements such as information provision and links during the purchase process.

[0064] This invention provides a system that allows plant growers to easily monitor the condition of their plants using devices such as smartphones and tablets. Users take photos of the plants they are growing with their devices and send them to a cloud server via a dedicated application. The devices used must have a camera function for taking photos and an internet connection.

[0065] The server executes an algorithm using machine learning techniques to analyze the received image data. This algorithm is built on platforms such as TENSORFLOW® and PyTorch and can recognize the type and condition of plants. Based on the analysis results, it identifies the type of plant and any problems it has, and generates specific advice regarding appropriate cultivation methods and material selection.

[0066] Based on images taken by the user, the server can suggest nitrogen deficiency based on yellowing leaves and provide appropriate advice. The server then provides relevant material information based on the advice and notifies the user via their device. This notification feature helps the user check material details and purchase them online if necessary.

[0067] As a concrete example, consider a case where a user takes a picture of a rose they are growing at home and sends it to the system. The server analyzes the image and diagnoses that the rose has symptoms of powdery mildew. As a countermeasure, advice is generated to use a fungicide that is effective against this disease. Product information and purchase links for the appropriate fungicide are also provided, allowing the user to take action quickly.

[0068] An example of a prompt using a generative AI model is, "How can I find out about the white spots on the plant I'm growing?" In this way, users can easily get advice on specific problems.

[0069] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0070] Step 1:

[0071] The user takes a picture of the plant they are growing with their device. The input is image data of the plant. The device takes this image and prepares to send it to a server in the cloud via the application. In this process, the image data is compressed and converted into a format suitable for transfer. The output is image data ready for transmission. Specifically, the user opens the camera app, points the camera at the plant, and presses the shutter button.

[0072] Step 2:

[0073] The device sends captured image data to a server in the cloud. The input is image data from the user's device. The image data is transmitted over the internet using data communication technology. The server receives this data and stores it for subsequent processing. The output is the completion of storing the image data on the server. Specific operations include verifying that the device is connected to the internet and executing a program to send the data.

[0074] Step 3:

[0075] The server analyzes the received image data. The input is image data of plants stored on the server. It executes an algorithm using machine learning techniques to identify the type, characteristics, and condition of the plants in the image. During this process, data processing such as feature extraction and pattern matching is performed, and the identification result is obtained as output. Specifically, the process involves extracting necessary information from the image data via a machine learning model.

[0076] Step 4:

[0077] The server generates specific advice on plant cultivation and management based on the identification results. The input is the result of image recognition. In this process, appropriate cultivation information is retrieved from the database based on the identification data, and advice is created using a generative AI model. The output is an advice document to be provided to the user. The specific operations include executing database queries and generating text using a generative AI algorithm.

[0078] Step 5:

[0079] The server presents recommended material information based on the advice and prepares purchase support information. The input is the generated advice. It retrieves recommended material information from the database and generates links so that the user can easily purchase it. The output is material information and purchase links. The specific operation includes the extraction of relevant data and the link generation process.

[0080] Step 6:

[0081] The terminal notifies the user of generated advice and recommended materials. The input is the advice and materials provided by the server. The terminal displays this on the screen and prompts the user for confirmation. The output is information that the user can visually understand. Specific actions include generating notifications, updating the display, and providing information to the user.

[0082] (Application Example 1)

[0083] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0084] In plant cultivation, it is crucial to accurately understand the plant's condition and take prompt and precise action. However, traditional methods require users to assess the plant's condition themselves and find appropriate solutions, which presents challenges if they lack the necessary knowledge and experience. Furthermore, there is a need for efficient methods to constantly monitor plant health and provide timely care.

[0085] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0086] In this invention, the server includes means for analyzing images of captured plants and executing an algorithm to identify the characteristics and condition of the plants; means for generating advice on how to cultivate the plants based on the identified characteristics and condition; means for presenting recommended materials in accordance with the generated advice and assisting in the purchase of those materials; and an operative unit equipped with a camera for automatically monitoring the condition of the plants and diagnosing their health, and which autonomously provides advice based on the diagnostic results. This enables the user to understand the condition of the plants and efficiently and accurately select appropriate cultivation methods and materials.

[0087] An "algorithm" is a set of computational steps or processing methods designed to solve a specific problem.

[0088] "Characteristics" refer to the individual properties or attributes that an object possesses, and are key elements in identification and classification.

[0089] "State" refers to the physical and functional condition or circumstances of an object at a specific point in time.

[0090] "Advice" refers to information that provides guidance or instruction regarding actions or choices to be taken in a particular situation.

[0091] "Materials" refers to items or materials used for a specific purpose or use, and in this context, it refers to items necessary for plant cultivation.

[0092] A "photography device" is a device equipped with sensors that capture light and image information, and in this context, it is used to record the condition of plants.

[0093] An "operator" is a mechanism or device that performs a specific action or function, and in this context, it refers to a component that operates for the purpose of monitoring and diagnosing plants.

[0094] This invention is a system for understanding the condition of plants and providing appropriate advice and materials. The system consists of an operating unit equipped with a camera, a cloud server, and a user terminal.

[0095] The server receives images of plants acquired by the camera and executes an image analysis algorithm. This algorithm is trained on a generative AI model and identifies the characteristics and condition of the plants. Based on the identified results, the server generates specific advice on cultivation methods and suggests appropriate materials. This information is transmitted to the user's terminal via the cloud environment.

[0096] The device notifies the user of the advice and recommended materials received. Based on the displayed information, the user can purchase materials online and care for their plants. The system also allows for continuous monitoring of the plant's condition and timely responses to changes in its health.

[0097] As a concrete example, a robot periodically takes pictures of a plant at a user's home to monitor it and sends them to a server. If the server analyzes the images and detects any abnormalities in the plant's leaves, detailed advice on the cause and countermeasures is sent to the user's device. For example, prompts such as "Please diagnose the health of this plant" or "There are black spots on the plant. How should I deal with this?" are used to facilitate more specific analysis.

[0098] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0099] Step 1:

[0100] The user's terminal issues commands to the operating device, and the camera acquires images of plants. In this process, the camera photographs the plants at specific angles and distances, and adjusts the light intensity appropriately to obtain clear images. The input is the plant's actual visual information, and the output is the acquired digital image data.

[0101] Step 2:

[0102] The terminal transmits the acquired digital image data to a cloud server. Here, data is transmitted efficiently and securely via an internet connection, facilitating rapid processing on the server side. The input is digital image data, and the output is a signal indicating completion of the image data transmission to the server.

[0103] Step 3:

[0104] The server executes an image analysis algorithm using a generative AI model on the received image data. This algorithm identifies the characteristics and condition of plants in the image and extracts the necessary information. The input is the transmitted digital image data, and the output is analysis result data regarding the characteristics and condition of the plants.

[0105] Step 4:

[0106] The server generates advice on plant cultivation methods based on the analysis results. Here, prompts are provided to an AI model to suggest appropriate countermeasures and care methods based on the current state of the plants. The input is the analysis result data, and the output is advice on specific cultivation methods.

[0107] Step 5:

[0108] The server selects the most relevant material information related to the advice it generates and sends it to the user's terminal. This material information includes details of recommended products and purchase links. Inputs are the advice text and the material database, while output is material suggestion information for user notification.

[0109] Step 6:

[0110] The terminal notifies the user of received advice and material information. Based on this, the user can properly care for their plants and purchase materials online as needed. The input is advice and material information from the server, and the output is the information notification to the user.

[0111] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0112] This invention provides a system that enables user interaction while considering emotions in order to enhance plant cultivation support. This system combines plant image analysis and emotion recognition functions to provide optimal advice and feedback tailored to the user's needs.

[0113] Users take photos of plants with their smartphones or tablets and provide emotional data. This emotional data is estimated from the user's voice tone and text input. The device sends the acquired plant images and emotional data to a server in the cloud. The server analyzes the images using a dedicated algorithm to identify the plant's characteristics and condition. Based on this information, it generates advice on how to care for the plants.

[0114] The server uses an emotion engine to analyze the user's emotional tendencies. For example, if the user is feeling anxious, the server generates advice that includes encouraging messages and helpful guides. On the other hand, if the user is confident, it may provide more detailed technical information.

[0115] In user notifications, the way advice is presented is adjusted based on information obtained from sentiment analysis. For example, the UI tone and message style are changed according to the user's emotions, creating a more user-friendly and accessible interface. A specific example is providing step-by-step instructions and simple illustrated guides when a user is emotionally confused.

[0116] Furthermore, the server generates a list of recommended materials based on the analysis results and provides emotionally tailored purchasing support. For example, for users who are feeling anxious, it simultaneously presents additional information to persuade them of the necessity of the purchase, as well as past success stories. This system allows users to cultivate plants more effectively while gaining emotional satisfaction.

[0117] The following describes the processing flow.

[0118] Step 1:

[0119] Users take photos of plants with their smartphones or tablets and save them to their devices using a dedicated application. When taking photos, paying attention to the angle at which the entire plant is visible and the amount of light will facilitate accurate analysis.

[0120] Step 2:

[0121] The device collects emotional data from the user. This emotional data can be obtained by recording and analyzing the user's voice messages, or by using text input. The obtained data is passed to emotion recognition software to estimate the user's emotional state.

[0122] Step 3:

[0123] The device sends image data of plants it has photographed, along with estimated emotion data, to a server in the cloud. This transmission uses necessary security protocols to protect the data.

[0124] Step 4:

[0125] The server executes an image analysis algorithm to analyze the submitted plant images. In this process, the server identifies the plant species and health condition, and extracts its characteristics. It also identifies disease conditions and environmental stresses that should be flagged.

[0126] Step 5:

[0127] The server uses an emotion engine to analyze the user's emotional data. For example, if the user is at ease, the server is configured to provide more detailed plant care information, and if they are anxious, it generates more supportive guidance.

[0128] Step 6:

[0129] The server generates advice based on the plant's analysis results and the user's emotional state. This advice includes specific plant care methods (nutrient supply, watering frequency, pest and disease control, etc.). It is also customized to include encouraging words and presentation methods appropriate to the user's situation.

[0130] Step 7:

[0131] The server generates a list of recommended materials and notifies the user. Based on sentiment data, it adds information to the user that emphasizes the effectiveness of the materials and facilitates the purchase process.

[0132] Step 8:

[0133] The device receives notifications from the server and presents them to the user in the most appropriate feedback format based on their emotional state. The user reviews the provided advice and recommended materials and purchases them online as needed. This allows the user to cultivate plants more effectively.

[0134] (Example 2)

[0135] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0136] Conventional plant cultivation support systems have struggled to provide effective advice that takes user emotions into consideration. Systems that only perform image analysis of plants lack feedback based on user emotions, and there was a need to improve user satisfaction with cultivation.

[0137] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0138] In this invention, the server includes means for analyzing images of captured plants and performing information processing to identify the characteristics and state of the organism; means for generating instructions on how to manage the organism based on the identified characteristics and state; and means for analyzing user emotion data and adjusting the instructions based on that data. This makes it possible to provide accurate advice that takes the user's emotions into consideration.

[0139] "Information processing" is the technology used to manipulate data and generate results that meet specific purposes.

[0140] The term "living organism" refers to all living things, including plants, that have the characteristic of growing and changing in response to their environment.

[0141] An "input / output device" is a hardware and software system that enables the transmission and reception of data.

[0142] "Identification" is the act of finding specific characteristics or patterns and clarifying their type or state.

[0143] A "visualization device" is a means of displaying information to a user visually, and usually includes displays and the like.

[0144] "Instructions" are pieces of information or commands provided to request a specific action or response.

[0145] "Emotional data" refers to information that indicates the user's emotional state, and is data estimated from voice and text input.

[0146] A "network environment" is an infrastructure that enables multiple computers and devices to communicate information.

[0147] "Acquisition" refers to the act of obtaining necessary data or information.

[0148] "Recommended materials" are tools or materials whose use is encouraged to achieve a specific purpose.

[0149] This invention is a system that efficiently supports plant cultivation and aims to realize user interaction that takes into account the user's emotions. The system combines plant image analysis and emotion recognition functions to provide the user with optimal cultivation advice.

[0150] Users take photos of plants using devices such as smartphones and tablets, and simultaneously input emotional data. This emotional data is provided via text or voice input. The device transmits this data to a cloud server via broadband communication. The transmitted data is typically packaged in JSON format using the HTTP protocol.

[0151] The server uses machine learning models for image analysis. This process employs software such as TensorFlow and OpenCV to identify the health status of plants based on the shape and color of their leaves. Furthermore, natural language processing techniques are used for sentiment analysis, utilizing libraries such as Python's NLTK library. This makes it possible to classify emotional tendencies based on user input into positive, negative, or neutral.

[0152] The server also generates cultivation advice based on the analysis results. Appropriate advice is created according to the analyzed state of the plant and the user's emotional tendencies. For example, if the plant is nutrient-deficient and the user is expressing anxiety, the server will provide detailed information on the type of fertilizer needed and how to use it, along with encouraging messages to alleviate the user's anxiety.

[0153] A concrete example of a prompt message would be, "I'm worried because my plant's leaves have recently turned yellow." Based on this input, the system analyzes the cause of the leaf discoloration and provides effective advice. By utilizing this system, users can cultivate their plants with a sense of emotional satisfaction.

[0154] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0155] Step 1:

[0156] The user takes a picture of a plant with their device and inputs sentiment data in text or voice. This process generates an image file of the plant and sentiment data as input data. The sentiment data is raw text or voice data based on the user's input.

[0157] Step 2:

[0158] The device sends the acquired image data and emotion data to a server in the cloud. The transmitted data is structured in JSON format and sent to the server over the network using the HTTP protocol. This process provides the server with the data necessary for analysis.

[0159] Step 3:

[0160] The server analyzes the received image data, using machine learning models to identify plant features. TensorFlow and OpenCV are used for image processing. The input data consists of plant images, and the output data is an identification result regarding the plant's health. For example, if the leaf color is abnormal, nutrient deficiency is identified as the cause.

[0161] Step 4:

[0162] The server analyzes sentiment data using natural language processing techniques to determine the user's emotional tendencies. In this step, the input text is parsed using the Python NLTK library. The input is the user's sentiment data, and the output is a classification of positive, negative, or neutral sentiment.

[0163] Step 5:

[0164] The server generates appropriate advice based on the plant's health and the user's emotional state. This process creates text messages that take into account the plant's condition and the user's emotions. The output is the advice provided to the user.

[0165] Step 6:

[0166] The server sends the generated advice and recommended materials information back to the terminal. The data is transmitted again via the HTTP protocol over the network and is ready to be displayed on the terminal. In this step, the user receives clear advice and information on recommended materials as needed.

[0167] (Application Example 2)

[0168] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0169] In recent years, home gardening has become increasingly popular, but many users struggle to accurately assess the health of their plants and implement appropriate gardening methods. Furthermore, there is a lack of interaction that considers user emotions, and psychological support is rarely provided. Additionally, obtaining appropriate gardening advice and related supplies is difficult.

[0170] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0171] In this invention, the server includes means for performing mathematical procedures to analyze images of captured plants and identify their characteristics and condition; means for generating advice on how to cultivate the plants based on the identified characteristics and condition; and means for analyzing the user's emotions from voice and text input and adjusting the content and method of providing the advice based on the analysis results. This allows the user to receive appropriate cultivation advice along with psychological support, and to easily obtain the necessary items for cultivation.

[0172] "Analyzing images of photographed plants" means processing the photographic data of plants taken by users using computer vision technology to understand the characteristics and condition of those plants.

[0173] "Identifying characteristics and conditions" means identifying information such as the type of plant, its health, and its growth stage based on the analysis results.

[0174] "Performing mathematical procedures" refers to the process of applying various algorithms to captured images and performing data analysis.

[0175] "Generating advice on cultivation methods" means creating suggestions for optimal cultivation methods and treatments based on the plant's condition obtained through analysis.

[0176] "Analyzing user emotions from voice and text input" means using emotion recognition technology to identify emotions from the tone of voice spoken by the user and the text entered.

[0177] "Adjusting the content and method of providing advice" refers to adjusting the method of feedback and messaging to users based on the results of sentiment analysis.

[0178] "Supporting the acquisition of goods" means recommending materials and products necessary for plant cultivation and providing users with information on how to acquire them.

[0179] The embodiment of this invention begins with a user taking a picture of a plant with a smartphone or tablet and providing associated emotional data. The user's device transmits the captured image, along with the acquired audio and text data, to a server in the cloud. This server is implemented by integrating the following functions.

[0180] First, the server uses computer vision technology to analyze images and identify the characteristics and condition of the plants. Specifically, it commonly uses software frameworks for image analysis such as OpenCV or TensorFlow. This allows it to identify the type of plant and its health status.

[0181] Next, emotion recognition technology is used to analyze emotions from voice and text provided by the user. This analysis employs techniques such as speech tone analysis and Hugging Face Transformers as part of natural language processing (NLP). Based on the obtained emotion data, the server generates optimal parenting advice tailored to the user's psychological state.

[0182] Furthermore, the system recommends items corresponding to the generated training advice to the user and notifies them with information to help them acquire those items. For example, a generating AI model in the cloud can provide specific training advice using AWS® Lambda or SageMaker.

[0183] For example, if a user feels that "this plant doesn't seem healthy," the system uses sentiment analysis to understand the user's anxiety and then provides advice such as, "This plant needs a little more water. Please water it this much next time."

[0184] Example of a prompt:

[0185] “Provide emotional and care advice for plant maintenance for a worried user, especially focusing on water requirements and general care tips.”

[0186] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0187] Step 1:

[0188] The user takes a picture of a plant with their smartphone and provides voice or text input. The input data consists of the plant image file and the user's voice or text data. The device collects this data and sends it to a cloud server when ready.

[0189] Step 2:

[0190] The server analyzes the received image data using computer vision software (e.g., OpenCV, TensorFlow). The input is image data of plants, and the output generates data that identifies the characteristics of the plants (e.g., species and health status). The server uses this analysis result to understand the condition of the plants.

[0191] Step 3:

[0192] The server analyzes the received audio or text data using an emotion recognition system (e.g., Hugging Face Transformers). The input is the user's audio file or text data, and the output identifies the user's emotional state (e.g., anxiety, confidence, etc.). The server uses this analysis to evaluate the user's psychological state.

[0193] Step 4:

[0194] The server generates cultivation advice using a generative AI model based on the analysis results of the plants and the user's emotional state. The inputs are plant condition data and user emotional data, and the output is specific advice to be provided to the user. The server adjusts the format of this advice according to the user's emotions.

[0195] Step 5:

[0196] Based on the generated advice, the server selects items recommended for the user and notifies the user, including this information. The input is the generated advice, and the output is notification data containing information about the recommended items and the advice. The server sends this data to the user's terminal for display.

[0197] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0198] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0199] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0200] [Second Embodiment]

[0201] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0202] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0203] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0204] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0205] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0206] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0207] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0208] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0209] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0210] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0211] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0212] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0213] This invention is a system that utilizes devices such as smartphones and tablets owned by plant growers to easily monitor the condition of plants and to support the selection of appropriate cultivation methods and materials.

[0214] The first step for the user is to take a picture of their plant using a smartphone or other device. The device sends this picture to a server in the cloud. The server then runs a dedicated algorithm based on the received image. This algorithm is pre-trained using machine learning techniques and can identify the type of plant, as well as the color, shape, and condition of its leaves and stems, through image analysis.

[0215] Based on the analysis results, the server identifies potential problems with the plant. For example, yellowing leaves may indicate a nitrogen deficiency, and unusual leaf shapes may suggest disease. Based on these analysis results, the server generates specific advice, including which nutrients are lacking and whether adjustments to sunlight or watering should be made.

[0216] Furthermore, the server provides information about the materials needed to solve the problem. This information includes details about the best materials for the plant type and the problem it is facing, such as specific fertilizers and soil amendments. The terminal notifies the user of this information and helps the user purchase the materials directly online.

[0217] As a concrete example, consider a scenario where a user takes a picture of their rose bush at home and sends it to the system. The server diagnoses that the rose is infected with powdery mildew and generates advice to use a fungicide effective against this disease. It also provides product information and online purchase links for the appropriate fungicide. This entire process allows the user to address plant problems quickly and effectively.

[0218] The following describes the processing flow.

[0219] Step 1:

[0220] Users take photos of plants using their smartphones or tablets. When taking photos, users pay attention to the angle and lighting so that the entire plant can be clearly seen.

[0221] Step 2:

[0222] The device receives the photographed plant image and adjusts the image size and resolution as needed. Next, the device sends the adjusted image data to a server in the cloud.

[0223] Step 3:

[0224] The server receives the image sent from the terminal. The server passes this image to an image analysis module, which uses a deep learning algorithm to analyze the characteristics and condition of the plants in the image. This analysis identifies the type of plant, leaf color and lesions, shape, and other characteristics.

[0225] Step 4:

[0226] The server evaluates the plant's health based on the analysis results. The evaluation process lists known problems (e.g., nutrient deficiencies, pest and disease effects) and determines whether those problems exist.

[0227] Step 5:

[0228] The server generates advice for the user based on the evaluation results. This advice includes things like watering, adjusting sunlight, and treating diseases.

[0229] Step 6:

[0230] The server lists recommended materials based on the plant type and condition. The list includes the specific name of the material, its effects, and a link to purchase it.

[0231] Step 7:

[0232] The device receives advice and recommended materials information sent from the server and notifies the user. The user can review the displayed information and purchase the recommended materials via the online store if necessary.

[0233] (Example 1)

[0234] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0235] One of the challenges faced by plant growers is the difficulty in accurately assessing plant health and quickly and effectively finding cultivation methods and problem-solving solutions. In particular, making expert judgments based on plant species and conditions, and providing cultivation advice and selecting materials accordingly, requires advanced knowledge and experience. Therefore, there is a need for methods that allow ordinary users to easily obtain this information and manage plant health.

[0236] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0237] In this invention, the server includes means for analyzing images of captured plants and executing an algorithm using machine learning techniques to identify the type of plant and the color, shape, and condition of its leaves and stems; means for automatically generating specific advice on how to cultivate and manage the plant based on the identified type, characteristics, and condition; and means including an interface that provides recommended material information based on the generated advice and assists in the online purchase of such materials. This makes it possible for users to easily understand the health of plants and quickly perform appropriate cultivation and problem solving without requiring specialized knowledge.

[0238] "Analyzing plant images" refers to processing plant image data transmitted from a terminal on a computer to identify the type of plant, as well as the color, shape, and condition of its leaves and stems.

[0239] An "algorithm using machine learning technology" is a technique for training computers to make accurate inferences based on vast amounts of data. In the context of plant image analysis, it refers to a program used to classify and recognize the features of image data based on a learned model.

[0240] "Specific advice on plant cultivation and management methods" refers to instructions and suggestions provided based on the analyzed state of the plant, including information on nutritional supplementation and environmental adjustments to maintain and improve plant health.

[0241] "Recommended materials information" refers to detailed information about products such as fertilizers, soil amendments, and fungicides that are recommended for use to address specific plant conditions or problems.

[0242] An "online purchase support interface" refers to a user interface that allows users to easily purchase recommended materials, and includes elements such as information provision and links during the purchase process.

[0243] This invention provides a system that allows plant growers to easily monitor the condition of their plants using devices such as smartphones and tablets. Users take photos of the plants they are growing with their devices and send them to a cloud server via a dedicated application. The devices used must have a camera function for taking photos and an internet connection.

[0244] The server executes an algorithm using machine learning techniques to analyze the received image data. This algorithm is built on platforms such as TensorFlow and PyTorch and can recognize the type and condition of plants. Based on the analysis results, it identifies the type of plant and any problems it has, and generates specific advice regarding appropriate cultivation methods and material selection.

[0245] Based on images taken by the user, the server can suggest nitrogen deficiency based on yellowing leaves and provide appropriate advice. The server then provides relevant material information based on the advice and notifies the user via their device. This notification feature helps the user check material details and purchase them online if necessary.

[0246] As a concrete example, consider a case where a user takes a picture of a rose they are growing at home and sends it to the system. The server analyzes the image and diagnoses that the rose has symptoms of powdery mildew. As a countermeasure, advice is generated to use a fungicide that is effective against this disease. Product information and purchase links for the appropriate fungicide are also provided, allowing the user to take action quickly.

[0247] An example of a prompt using a generative AI model is, "How can I find out about the white spots on the plant I'm growing?" In this way, users can easily obtain advice on specific problems.

[0248] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0249] Step 1:

[0250] The user takes a picture of the plant they are growing with their device. The input is image data of the plant. The device takes this image and prepares to send it to a server in the cloud via the application. In this process, the image data is compressed and converted into a format suitable for transfer. The output is image data ready for transmission. Specifically, the user opens the camera app, points the camera at the plant, and presses the shutter button.

[0251] Step 2:

[0252] The device sends captured image data to a server in the cloud. The input is image data from the user's device. The image data is transmitted over the internet using data communication technology. The server receives this data and stores it for subsequent processing. The output is the completion of storing the image data on the server. Specific operations include verifying that the device is connected to the internet and executing a program to send the data.

[0253] Step 3:

[0254] The server analyzes the received image data. The input is image data of plants stored on the server. It executes an algorithm using machine learning techniques to identify the type, characteristics, and condition of the plants in the image. During this process, data processing such as feature extraction and pattern matching is performed, and the identification result is obtained as output. Specifically, the process involves extracting necessary information from the image data via a machine learning model.

[0255] Step 4:

[0256] The server generates specific advice on plant cultivation and management based on the identification results. The input is the result of image recognition. In this process, appropriate cultivation information is retrieved from the database based on the identification data, and advice is created using a generative AI model. The output is an advice document to be provided to the user. The specific operations include executing database queries and generating text using a generative AI algorithm.

[0257] Step 5:

[0258] The server presents recommended material information based on the advice and prepares purchase support information. The input is the generated advice. It retrieves recommended material information from the database and generates links so that the user can easily purchase it. The output is material information and purchase links. The specific operation includes the extraction of relevant data and the link generation process.

[0259] Step 6:

[0260] The terminal notifies the user of generated advice and recommended materials. The input is the advice and materials provided by the server. The terminal displays this on the screen and prompts the user for confirmation. The output is information that the user can visually understand. Specific actions include generating notifications, updating the display, and providing information to the user.

[0261] (Application Example 1)

[0262] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] In plant cultivation, it is crucial to accurately understand the plant's condition and take prompt and precise action. However, traditional methods require users to assess the plant's condition themselves and find appropriate solutions, which presents challenges if they lack the necessary knowledge and experience. Furthermore, there is a need for efficient methods to constantly monitor plant health and provide timely care.

[0264] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0265] In this invention, the server includes means for analyzing images of captured plants and executing an algorithm to identify the characteristics and condition of the plants; means for generating advice on how to cultivate the plants based on the identified characteristics and condition; means for presenting recommended materials in accordance with the generated advice and assisting in the purchase of those materials; and an operative unit equipped with a camera for automatically monitoring the condition of the plants and diagnosing their health, and which autonomously provides advice based on the diagnostic results. This enables the user to understand the condition of the plants and efficiently and accurately select appropriate cultivation methods and materials.

[0266] An "algorithm" is a set of computational steps or processing methods designed to solve a specific problem.

[0267] "Characteristics" refer to the individual properties or attributes that an object possesses, and are key elements in identification and classification.

[0268] "State" refers to the physical and functional condition or circumstances of an object at a specific point in time.

[0269] "Advice" refers to information that provides guidance or instruction regarding actions or choices to be taken in a particular situation.

[0270] "Materials" refers to items or materials used for a specific purpose or use, and in this context, it refers to items necessary for plant cultivation.

[0271] A "photography device" is a device equipped with sensors that capture light and image information, and in this context, it is used to record the condition of plants.

[0272] An "operator" is a mechanism or device that performs a specific action or function, and in this context, it refers to a component that operates for the purpose of monitoring and diagnosing plants.

[0273] This invention is a system for understanding the condition of plants and providing appropriate advice and materials. The system consists of an operating unit equipped with a camera, a cloud server, and a user terminal.

[0274] The server receives images of plants acquired by the camera and executes an image analysis algorithm. This algorithm is trained on a generative AI model and identifies the characteristics and condition of the plants. Based on the identified results, the server generates specific advice on cultivation methods and suggests appropriate materials. This information is transmitted to the user's terminal via the cloud environment.

[0275] The device notifies the user of the advice and recommended materials received. Based on the displayed information, the user can purchase materials online and care for their plants. The system also allows for continuous monitoring of the plant's condition and timely responses to changes in its health.

[0276] As a specific example, a robot periodically takes pictures of a plant at home for a user to monitor and sends them to a server. If an abnormality is found in the plant's leaves as a result of analysis on the server, detailed advice on the cause and countermeasures will be notified to the user's terminal. For example, more specific analysis is promoted by using prompt sentences such as "Please diagnose the health condition of this plant" or "There are black spots on the plant. How should I deal with them?"

[0277] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0278] Step 1:

[0279] The user's terminal gives an instruction to the operating body, and the imaging device acquires an image of the plant. At this time, the camera takes a picture of the plant at a specific angle and distance, and appropriately adjusts the light amount to obtain a clear image. The input is the actual visual information of the plant, and the output is the acquired digital image data.

[0280] Step 2:

[0281] The terminal transmits the acquired digital image data to the cloud server. Here, the data is transmitted efficiently and securely via an Internet connection to facilitate rapid processing on the server side. The input is the digital image data, and the output is a signal indicating the completion of the transmission of the image data to the server.

[0282] Step 3:

[0283] For the image data received by the server, an image analysis algorithm using a generative AI model is executed. This algorithm identifies the characteristics and states of the plants in the image and extracts the necessary information. The input is the transmitted digital image data, and the output is the analysis result data regarding the characteristics and states of the plants.

[0284] Step 4:

[0285] Based on the analysis results, the server generates advice on plant cultivation methods. Here, a prompt sentence for proposing appropriate countermeasures and maintenance methods based on the current situation of the plant is given to the generation AI model. The input is the analysis result data, and the output is an advice sentence on specific cultivation methods.

[0286] Step 5:

[0287] The server selects the optimal material information related to the advice it generated and sends it to the user's terminal. The material information includes details of the recommended products and purchase links. The input is the advice sentence and the material database, and the output is the material proposal information for notifying the user.

[0288] Step 6:

[0289] The terminal notifies the user of the received advice and material information. Based on this, the user can perform appropriate plant maintenance and purchase materials online if necessary. The input is the advice and material information from the server, and the output is the information notification to the user.

[0290] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0291] The present invention is a system that enables interaction considering the user's emotion in order to enhance plant cultivation support. This system combines a plant image analysis function and an emotion recognition function to provide optimal advice and feedback according to the user's needs.

[0292] Users take photos of plants with their smartphones or tablets and provide emotional data. This emotional data is estimated from the user's voice tone and text input. The device sends the acquired plant images and emotional data to a server in the cloud. The server analyzes the images using a dedicated algorithm to identify the plant's characteristics and condition. Based on this information, it generates advice on how to care for the plants.

[0293] The server uses an emotion engine to analyze the user's emotional tendencies. For example, if the user is feeling anxious, the server generates advice that includes encouraging messages and helpful guides. On the other hand, if the user is confident, it may provide more detailed technical information.

[0294] In user notifications, the way advice is presented is adjusted based on information obtained from sentiment analysis. For example, the UI tone and message style are changed according to the user's emotions, creating a more user-friendly and accessible interface. A specific example is providing step-by-step instructions and simple illustrated guides when a user is emotionally confused.

[0295] Furthermore, the server generates a list of recommended materials based on the analysis results and provides emotionally tailored purchasing support. For example, for users who are feeling anxious, it simultaneously presents additional information to persuade them of the necessity of the purchase, as well as past success stories. This system allows users to cultivate plants more effectively while gaining emotional satisfaction.

[0296] The following describes the processing flow.

[0297] Step 1:

[0298] Users take photos of plants with their smartphones or tablets and save them to their devices using a dedicated application. When taking photos, paying attention to the angle at which the entire plant is visible and the amount of light will facilitate accurate analysis.

[0299] Step 2:

[0300] The terminal collects emotional data from the user. The emotional data can be obtained by recording and analyzing the user's voice message or by using text input. The obtained data is passed to the emotion recognition software to estimate the user's emotional state.

[0301] Step 3:

[0302] The terminal transmits the image data of the plant captured and the estimated emotional data to the server on the cloud. For this transmission, the data is protected using the necessary security protocols.

[0303] Step 4:

[0304] The server executes an image analysis algorithm to analyze the transmitted image of the plant. In this process, the server identifies the type and health status of the plant, extracts features, and also identifies the disease conditions and environmental stresses that should be flagged.

[0305] Step 5:

[0306] The server uses an emotion engine to analyze the user's emotional data. The server is set, for example, to provide more detailed plant cultivation information when the user is at ease and to generate guidance that places more emphasis on support when the user is anxious.

[0307] Step 6:

[0308] The server generates advice according to the analysis results of the plant and the user's emotional state. This advice includes specific plant care methods (such as nutrient supply, watering frequency, pest control, etc.). It is also customized to include words of encouragement suitable for the user's situation and presentation methods suitable for the situation.

[0309] Step 7:

[0310] The server generates a list of recommended materials and notifies the user. Based on sentiment data, it adds information to the user that emphasizes the effectiveness of the materials and facilitates the purchase process.

[0311] Step 8:

[0312] The device receives notifications from the server and presents them to the user in the most appropriate feedback format based on their emotional state. The user reviews the provided advice and recommended materials and purchases them online as needed. This allows the user to cultivate plants more effectively.

[0313] (Example 2)

[0314] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0315] Conventional plant cultivation support systems have struggled to provide effective advice that takes user emotions into consideration. Systems that only perform image analysis of plants lack feedback based on user emotions, and there was a need to improve user satisfaction with cultivation.

[0316] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0317] In this invention, the server includes means for analyzing images of captured plants and performing information processing to identify the characteristics and state of the organism; means for generating instructions on how to manage the organism based on the identified characteristics and state; and means for analyzing user emotion data and adjusting the instructions based on that data. This makes it possible to provide accurate advice that takes the user's emotions into consideration.

[0318] "Information processing" is the technology used to manipulate data and generate results that meet specific purposes.

[0319] The term "living organism" refers to all living things, including plants, that have the characteristic of growing and changing in response to their environment.

[0320] An "input / output device" is a hardware and software system that enables the transmission and reception of data.

[0321] "Identification" is the act of finding specific characteristics or patterns and clarifying their type or state.

[0322] A "visualization device" is a means of displaying information to a user visually, and usually includes displays and the like.

[0323] "Instructions" are pieces of information or commands provided to request a specific action or response.

[0324] "Emotional data" refers to information that indicates the user's emotional state, and is data estimated from voice and text input.

[0325] A "network environment" is an infrastructure that enables multiple computers and devices to communicate information.

[0326] "Acquisition" refers to the act of obtaining necessary data or information.

[0327] "Recommended materials" are tools or materials whose use is encouraged to achieve a specific purpose.

[0328] This invention is a system that efficiently supports plant cultivation and aims to realize user interaction that takes into account the user's emotions. The system combines plant image analysis and emotion recognition functions to provide the user with optimal cultivation advice.

[0329] Users take photos of plants using devices such as smartphones and tablets, and simultaneously input emotional data. This emotional data is provided via text or voice input. The device transmits this data to a cloud server via broadband communication. The transmitted data is typically packaged in JSON format using the HTTP protocol.

[0330] The server uses machine learning models for image analysis. This process employs software such as TensorFlow and OpenCV to identify the health status of plants based on the shape and color of their leaves. Furthermore, natural language processing techniques are used for sentiment analysis, utilizing libraries such as Python's NLTK library. This makes it possible to classify emotional tendencies based on user input into positive, negative, or neutral.

[0331] The server also generates cultivation advice based on the analysis results. Appropriate advice is created according to the analyzed state of the plant and the user's emotional tendencies. For example, if the plant is nutrient-deficient and the user is expressing anxiety, the server will provide detailed information on the type of fertilizer needed and how to use it, along with encouraging messages to alleviate the user's anxiety.

[0332] A concrete example of a prompt message would be, "I'm worried because my plant's leaves have recently turned yellow." Based on this input, the system analyzes the cause of the leaf discoloration and provides effective advice. By utilizing this system, users can cultivate their plants with a sense of emotional satisfaction.

[0333] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0334] Step 1:

[0335] The user takes a picture of a plant with their device and inputs sentiment data in text or voice. This process generates an image file of the plant and sentiment data as input data. The sentiment data is raw text or voice data based on the user's input.

[0336] Step 2:

[0337] The device sends the acquired image data and emotion data to a server in the cloud. The transmitted data is structured in JSON format and sent to the server over the network using the HTTP protocol. This process provides the server with the data necessary for analysis.

[0338] Step 3:

[0339] The server analyzes the received image data, using machine learning models to identify plant features. TensorFlow and OpenCV are used for image processing. The input data consists of plant images, and the output data is an identification result regarding the plant's health. For example, if the leaf color is abnormal, nutrient deficiency is identified as the cause.

[0340] Step 4:

[0341] The server analyzes sentiment data using natural language processing techniques to determine the user's emotional tendencies. In this step, the input text is parsed using the Python NLTK library. The input is the user's sentiment data, and the output is a classification of positive, negative, or neutral sentiment.

[0342] Step 5:

[0343] The server generates appropriate advice based on the plant's health and the user's emotional state. This process creates text messages that take into account the plant's condition and the user's emotions. The output is the advice provided to the user.

[0344] Step 6:

[0345] The server sends the generated advice and recommended materials information back to the terminal. The data is transmitted again via the HTTP protocol over the network and is ready to be displayed on the terminal. In this step, the user receives clear advice and information on recommended materials as needed.

[0346] (Application Example 2)

[0347] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0348] In recent years, home gardening has become increasingly popular, but many users struggle to accurately assess the health of their plants and implement appropriate gardening methods. Furthermore, there is a lack of interaction that considers user emotions, and psychological support is rarely provided. Additionally, obtaining appropriate gardening advice and related supplies is difficult.

[0349] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0350] In this invention, the server includes means for performing mathematical procedures to analyze images of captured plants and identify their characteristics and condition; means for generating advice on how to cultivate the plants based on the identified characteristics and condition; and means for analyzing the user's emotions from voice and text input and adjusting the content and method of providing the advice based on the analysis results. This allows the user to receive appropriate cultivation advice along with psychological support, and to easily obtain the necessary items for cultivation.

[0351] "Analyzing images of photographed plants" means processing the photographic data of plants taken by users using computer vision technology to understand the characteristics and condition of those plants.

[0352] "Identifying characteristics and conditions" means identifying information such as the type of plant, its health, and its growth stage based on the analysis results.

[0353] "Performing mathematical procedures" refers to the process of applying various algorithms to captured images and performing data analysis.

[0354] "Generating advice on cultivation methods" means creating suggestions for optimal cultivation methods and treatments based on the plant's condition obtained through analysis.

[0355] "Analyzing user emotions from voice and text input" means using emotion recognition technology to identify emotions from the tone of voice spoken by the user and the text entered.

[0356] "Adjusting the content and method of providing advice" refers to adjusting the method of feedback and messaging to users based on the results of sentiment analysis.

[0357] "Supporting the acquisition of goods" means recommending materials and products necessary for plant cultivation and providing users with information on how to acquire them.

[0358] The embodiment of this invention begins with a user taking a picture of a plant with a smartphone or tablet and providing associated emotional data. The user's device transmits the captured image, along with the acquired audio and text data, to a server in the cloud. This server is implemented by integrating the following functions.

[0359] First, the server uses computer vision technology to analyze images and identify the characteristics and condition of the plants. Specifically, it commonly uses software frameworks for image analysis such as OpenCV or TensorFlow. This allows it to identify the type of plant and its health status.

[0360] Next, emotion recognition technology is used to analyze emotions from voice and text provided by the user. This analysis employs techniques such as speech tone analysis and Hugging Face Transformers as part of natural language processing (NLP). Based on the obtained emotion data, the server generates optimal parenting advice tailored to the user's psychological state.

[0361] Furthermore, the system recommends items corresponding to the generated training advice to the user and notifies them with information to help them acquire those items. For example, AWS Lambda or SageMaker can be used to have a cloud-based AI model provide specific training advice.

[0362] For example, if a user feels that "this plant doesn't seem healthy," the system uses sentiment analysis to understand the user's anxiety and then provides advice such as, "This plant needs a little more water. Please water it this much next time."

[0363] Example of a prompt:

[0364] "Provide emotional and care advice for plant maintenance for a worried user, especially focusing on water requirements and general care tips."

[0365] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0366] Step 1:

[0367] The user takes a picture of a plant with their smartphone and provides voice or text input. The input data consists of the plant image file and the user's voice or text data. The device collects this data and sends it to a cloud server when ready.

[0368] Step 2:

[0369] The server analyzes the received image data using computer vision software (e.g., OpenCV, TensorFlow). The input is image data of plants, and the output generates data that identifies the characteristics of the plants (e.g., species and health status). The server uses this analysis result to understand the condition of the plants.

[0370] Step 3:

[0371] The server analyzes the received audio or text data using an emotion recognition system (e.g., Hugging Face Transformers). The input is the user's audio file or text data, and the output identifies the user's emotional state (e.g., anxiety, confidence, etc.). The server uses this analysis to evaluate the user's psychological state.

[0372] Step 4:

[0373] The server generates cultivation advice using a generative AI model based on the analysis results of the plants and the user's emotional state. The inputs are plant condition data and user emotional data, and the output is specific advice to be provided to the user. The server adjusts the format of this advice according to the user's emotions.

[0374] Step 5:

[0375] Based on the generated advice, the server selects items recommended for the user and notifies the user, including this information. The input is the generated advice, and the output is notification data containing information about the recommended items and the advice. The server sends this data to the user's terminal for display.

[0376] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0377] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0378] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0379] [Third Embodiment]

[0380] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0381] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0382] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0383] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0384] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0385] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0386] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0387] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0388] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0389] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0390] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0391] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0392] This invention is a system that utilizes devices such as smartphones and tablets owned by plant growers to easily monitor the condition of plants and to support the selection of appropriate cultivation methods and materials.

[0393] The first step for the user is to take a picture of their plant using a smartphone or other device. The device sends this picture to a server in the cloud. The server then runs a dedicated algorithm based on the received image. This algorithm is pre-trained using machine learning techniques and can identify the type of plant, as well as the color, shape, and condition of its leaves and stems, through image analysis.

[0394] Based on the analysis results, the server identifies potential problems with the plant. For example, yellowing leaves may indicate a nitrogen deficiency, and unusual leaf shapes may suggest disease. Based on these analysis results, the server generates specific advice, including which nutrients are lacking and whether adjustments to sunlight or watering should be made.

[0395] Furthermore, the server provides information about the materials needed to solve the problem. This information includes details about the best materials for the plant type and the problem it is facing, such as specific fertilizers and soil amendments. The terminal notifies the user of this information and helps the user purchase the materials directly online.

[0396] As a concrete example, consider a scenario where a user takes a picture of their rose bush at home and sends it to the system. The server diagnoses that the rose is infected with powdery mildew and generates advice to use a fungicide effective against this disease. It also provides product information and online purchase links for the appropriate fungicide. This entire process allows the user to address plant problems quickly and effectively.

[0397] The following describes the processing flow.

[0398] Step 1:

[0399] Users take photos of plants using their smartphones or tablets. When taking photos, users pay attention to the angle and lighting so that the entire plant can be clearly seen.

[0400] Step 2:

[0401] The device receives the photographed plant image and adjusts the image size and resolution as needed. Next, the device sends the adjusted image data to a server in the cloud.

[0402] Step 3:

[0403] The server receives the image sent from the terminal. The server passes this image to an image analysis module, which uses a deep learning algorithm to analyze the characteristics and condition of the plants in the image. This analysis identifies the type of plant, leaf color and lesions, shape, and other characteristics.

[0404] Step 4:

[0405] The server evaluates the plant's health based on the analysis results. The evaluation process lists known problems (e.g., nutrient deficiencies, pest and disease effects) and determines whether those problems exist.

[0406] Step 5:

[0407] The server generates advice for the user based on the evaluation results. This advice includes things like watering, adjusting sunlight, and treating diseases.

[0408] Step 6:

[0409] The server lists recommended materials based on the plant type and condition. The list includes the specific name of the material, its effects, and a link to purchase it.

[0410] Step 7:

[0411] The device receives advice and recommended materials information sent from the server and notifies the user. The user can review the displayed information and purchase the recommended materials via the online store if necessary.

[0412] (Example 1)

[0413] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0414] One of the challenges faced by plant growers is the difficulty in accurately assessing plant health and quickly and effectively finding cultivation methods and problem-solving solutions. In particular, making expert judgments based on plant species and conditions, and providing cultivation advice and selecting materials accordingly, requires advanced knowledge and experience. Therefore, there is a need for methods that allow ordinary users to easily obtain this information and manage plant health.

[0415] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0416] In this invention, the server includes means for analyzing images of captured plants and executing an algorithm using machine learning techniques to identify the type of plant and the color, shape, and condition of its leaves and stems; means for automatically generating specific advice on how to cultivate and manage the plant based on the identified type, characteristics, and condition; and means including an interface that provides recommended material information based on the generated advice and assists in the online purchase of such materials. This makes it possible for users to easily understand the health of plants and quickly perform appropriate cultivation and problem solving without requiring specialized knowledge.

[0417] "Analyzing plant images" refers to processing plant image data transmitted from a terminal on a computer to identify the type of plant, as well as the color, shape, and condition of its leaves and stems.

[0418] An "algorithm using machine learning technology" is a technique for training computers to make accurate inferences based on vast amounts of data. In the context of plant image analysis, it refers to a program used to classify and recognize the features of image data based on a learned model.

[0419] "Specific advice on plant cultivation and management methods" refers to instructions and suggestions provided based on the analyzed state of the plant, including information on nutritional supplementation and environmental adjustments to maintain and improve plant health.

[0420] "Recommended materials information" refers to detailed information about products such as fertilizers, soil amendments, and fungicides that are recommended for use to address specific plant conditions or problems.

[0421] An "online purchase support interface" refers to a user interface that allows users to easily purchase recommended materials, and includes elements such as information provision and links during the purchase process.

[0422] This invention provides a system that allows plant growers to easily monitor the condition of their plants using devices such as smartphones and tablets. Users take photos of the plants they are growing with their devices and send them to a cloud server via a dedicated application. The devices used must have a camera function for taking photos and an internet connection.

[0423] The server executes an algorithm using machine learning techniques to analyze the received image data. This algorithm is built on platforms such as TensorFlow and PyTorch and can recognize the type and condition of plants. Based on the analysis results, it identifies the type of plant and any problems it has, and generates specific advice regarding appropriate cultivation methods and material selection.

[0424] Based on images taken by the user, the server can suggest nitrogen deficiency based on yellowing leaves and provide appropriate advice. The server then provides relevant material information based on the advice and notifies the user via their device. This notification feature helps the user check material details and purchase them online if necessary.

[0425] As a concrete example, consider a case where a user takes a picture of a rose they are growing at home and sends it to the system. The server analyzes the image and diagnoses that the rose has symptoms of powdery mildew. As a countermeasure, advice is generated to use a fungicide that is effective against this disease. Product information and purchase links for the appropriate fungicide are also provided, allowing the user to take action quickly.

[0426] An example of a prompt using a generative AI model is, "How can I find out about the white spots on the plant I'm growing?" In this way, users can easily obtain advice on specific problems.

[0427] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0428] Step 1:

[0429] The user takes a picture of the plant they are growing with their device. The input is image data of the plant. The device takes this image and prepares to send it to a server in the cloud via the application. In this process, the image data is compressed and converted into a format suitable for transfer. The output is image data ready for transmission. Specifically, the user opens the camera app, points the camera at the plant, and presses the shutter button.

[0430] Step 2:

[0431] The device sends captured image data to a server in the cloud. The input is image data from the user's device. The image data is transmitted over the internet using data communication technology. The server receives this data and stores it for subsequent processing. The output is the completion of storing the image data on the server. Specific operations include verifying that the device is connected to the internet and executing a program to send the data.

[0432] Step 3:

[0433] The server analyzes the received image data. The input is image data of plants stored on the server. It executes an algorithm using machine learning techniques to identify the type, characteristics, and condition of the plants in the image. During this process, data processing such as feature extraction and pattern matching is performed, and the identification result is obtained as output. Specifically, the process involves extracting necessary information from the image data via a machine learning model.

[0434] Step 4:

[0435] The server generates specific advice on plant cultivation and management based on the identification results. The input is the result of image recognition. In this process, appropriate cultivation information is retrieved from the database based on the identification data, and advice is created using a generative AI model. The output is an advice document to be provided to the user. The specific operations include executing database queries and generating text using a generative AI algorithm.

[0436] Step 5:

[0437] The server presents recommended material information based on the advice and prepares purchase support information. The input is the generated advice. It retrieves recommended material information from the database and generates links so that the user can easily purchase it. The output is material information and purchase links. The specific operation includes the extraction of relevant data and the link generation process.

[0438] Step 6:

[0439] The terminal notifies the user of generated advice and recommended materials. The input is the advice and materials provided by the server. The terminal displays this on the screen and prompts the user for confirmation. The output is information that the user can visually understand. Specific actions include generating notifications, updating the display, and providing information to the user.

[0440] (Application Example 1)

[0441] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0442] In plant cultivation, it is crucial to accurately understand the plant's condition and take prompt and precise action. However, traditional methods require users to assess the plant's condition themselves and find appropriate solutions, which presents challenges if they lack the necessary knowledge and experience. Furthermore, there is a need for efficient methods to constantly monitor plant health and provide timely care.

[0443] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0444] In this invention, the server includes means for analyzing images of captured plants and executing an algorithm to identify the characteristics and condition of the plants; means for generating advice on how to cultivate the plants based on the identified characteristics and condition; means for presenting recommended materials in accordance with the generated advice and assisting in the purchase of those materials; and an operative unit equipped with a camera for automatically monitoring the condition of the plants and diagnosing their health, and which autonomously provides advice based on the diagnostic results. This enables the user to understand the condition of the plants and efficiently and accurately select appropriate cultivation methods and materials.

[0445] An "algorithm" is a set of computational steps or processing methods designed to solve a specific problem.

[0446] "Characteristics" refer to the individual properties or attributes that an object possesses, and are key elements in identification and classification.

[0447] "State" refers to the physical and functional condition or circumstances of an object at a specific point in time.

[0448] "Advice" refers to information that provides guidance or instruction regarding actions or choices to be taken in a particular situation.

[0449] "Materials" refers to items or materials used for a specific purpose or use, and in this context, it refers to items necessary for plant cultivation.

[0450] A "photography device" is a device equipped with sensors that capture light and image information, and in this context, it is used to record the condition of plants.

[0451] An "operator" is a mechanism or device that performs a specific action or function, and in this context, it refers to a component that operates for the purpose of monitoring and diagnosing plants.

[0452] This invention is a system for understanding the condition of plants and providing appropriate advice and materials. The system consists of an operating unit equipped with a camera, a cloud server, and a user terminal.

[0453] The server receives images of plants acquired by the camera and executes an image analysis algorithm. This algorithm is trained on a generative AI model and identifies the characteristics and condition of the plants. Based on the identified results, the server generates specific advice on cultivation methods and suggests appropriate materials. This information is transmitted to the user's terminal via the cloud environment.

[0454] The device notifies the user of the advice and recommended materials received. Based on the displayed information, the user can purchase materials online and care for their plants. The system also allows for continuous monitoring of the plant's condition and timely responses to changes in its health.

[0455] As a concrete example, a robot periodically takes pictures of a plant at a user's home to monitor it and sends them to a server. If the server analyzes the images and detects any abnormalities in the plant's leaves, detailed advice on the cause and countermeasures is sent to the user's device. For example, prompts such as "Please diagnose the health of this plant" or "There are black spots on the plant. How should I deal with this?" are used to facilitate more specific analysis.

[0456] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0457] Step 1:

[0458] The user's terminal issues commands to the operating device, and the camera acquires images of plants. In this process, the camera photographs the plants at specific angles and distances, and adjusts the light intensity appropriately to obtain clear images. The input is the plant's actual visual information, and the output is the acquired digital image data.

[0459] Step 2:

[0460] The terminal transmits the acquired digital image data to a cloud server. Here, data is transmitted efficiently and securely via an internet connection, facilitating rapid processing on the server side. The input is digital image data, and the output is a signal indicating completion of the image data transmission to the server.

[0461] Step 3:

[0462] The server executes an image analysis algorithm using a generative AI model on the received image data. This algorithm identifies the characteristics and condition of plants in the image and extracts the necessary information. The input is the transmitted digital image data, and the output is analysis result data regarding the characteristics and condition of the plants.

[0463] Step 4:

[0464] The server generates advice on plant cultivation methods based on the analysis results. Here, prompts are provided to an AI model to suggest appropriate countermeasures and care methods based on the current state of the plants. The input is the analysis result data, and the output is advice on specific cultivation methods.

[0465] Step 5:

[0466] The server selects the most relevant material information related to the advice it generates and sends it to the user's terminal. This material information includes details of recommended products and purchase links. Inputs are the advice text and the material database, while output is material suggestion information for user notification.

[0467] Step 6:

[0468] The terminal notifies the user of received advice and material information. Based on this, the user can properly care for their plants and purchase materials online as needed. The input is advice and material information from the server, and the output is the information notification to the user.

[0469] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0470] This invention provides a system that enables user interaction while considering emotions in order to enhance plant cultivation support. This system combines plant image analysis and emotion recognition functions to provide optimal advice and feedback tailored to the user's needs.

[0471] Users take photos of plants with their smartphones or tablets and provide emotional data. This emotional data is estimated from the user's voice tone and text input. The device sends the acquired plant images and emotional data to a server in the cloud. The server analyzes the images using a dedicated algorithm to identify the plant's characteristics and condition. Based on this information, it generates advice on how to care for the plants.

[0472] The server uses an emotion engine to analyze the user's emotional tendencies. For example, if the user is feeling anxious, the server generates advice that includes encouraging messages and helpful guides. On the other hand, if the user is confident, it may provide more detailed technical information.

[0473] In user notifications, the way advice is presented is adjusted based on information obtained from sentiment analysis. For example, the UI tone and message style are changed according to the user's emotions, creating a more user-friendly and accessible interface. A specific example is providing step-by-step instructions and simple illustrated guides when a user is emotionally confused.

[0474] Furthermore, the server generates a list of recommended materials based on the analysis results and provides emotionally tailored purchasing support. For example, for users who are feeling anxious, it simultaneously presents additional information to persuade them of the necessity of the purchase, as well as past success stories. This system allows users to cultivate plants more effectively while gaining emotional satisfaction.

[0475] The following describes the processing flow.

[0476] Step 1:

[0477] Users take photos of plants with their smartphones or tablets and save them to their devices using a dedicated application. When taking photos, paying attention to the angle at which the entire plant is visible and the amount of light will facilitate accurate analysis.

[0478] Step 2:

[0479] The device collects emotional data from the user. This emotional data can be obtained by recording and analyzing the user's voice messages, or by using text input. The obtained data is passed to emotion recognition software to estimate the user's emotional state.

[0480] Step 3:

[0481] The device sends image data of plants it has photographed, along with estimated emotion data, to a server in the cloud. This transmission uses necessary security protocols to protect the data.

[0482] Step 4:

[0483] The server executes an image analysis algorithm to analyze the submitted plant images. In this process, the server identifies the plant species and health status, and extracts its characteristics. It also identifies disease conditions and environmental stresses that should be flagged.

[0484] Step 5:

[0485] The server uses an emotion engine to analyze the user's emotional data. For example, if the user is at ease, the server is configured to provide more detailed plant care information, and if they are anxious, it generates more supportive guidance.

[0486] Step 6:

[0487] The server generates advice based on the plant's analysis results and the user's emotional state. This advice includes specific plant care methods (nutrient supply, watering frequency, pest and disease control, etc.). It is also customized to include encouraging words and presentation methods appropriate to the user's situation.

[0488] Step 7:

[0489] The server generates a list of recommended materials and notifies the user. Based on sentiment data, it adds information to the user that emphasizes the effectiveness of the materials and facilitates the purchase process.

[0490] Step 8:

[0491] The device receives notifications from the server and presents them to the user in the most appropriate feedback format based on their emotional state. The user reviews the provided advice and recommended materials and purchases them online as needed. This allows the user to cultivate plants more effectively.

[0492] (Example 2)

[0493] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0494] Conventional plant cultivation support systems have struggled to provide effective advice that takes user emotions into consideration. Systems that only perform image analysis of plants lack feedback based on user emotions, and there was a need to improve user satisfaction with cultivation.

[0495] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0496] In this invention, the server includes means for analyzing images of captured plants and performing information processing to identify the characteristics and state of the organism; means for generating instructions on how to manage the organism based on the identified characteristics and state; and means for analyzing user emotion data and adjusting the instructions based on that data. This makes it possible to provide accurate advice that takes the user's emotions into consideration.

[0497] "Information processing" is the technology used to manipulate data and generate results that meet specific purposes.

[0498] The term "living organism" refers to all living things, including plants, that have the characteristic of growing and changing in response to their environment.

[0499] An "input / output device" is a hardware and software system that enables the transmission and reception of data.

[0500] "Identification" is the act of finding specific characteristics or patterns and clarifying their type or state.

[0501] A "visualization device" is a means of displaying information to a user visually, and usually includes displays and the like.

[0502] "Instructions" are pieces of information or commands provided to request a specific action or response.

[0503] "Emotional data" refers to information that indicates the user's emotional state, and is data estimated from voice and text input.

[0504] A "network environment" is an infrastructure that enables multiple computers and devices to communicate information.

[0505] "Acquisition" refers to the act of obtaining necessary data or information.

[0506] "Recommended materials" are tools or materials whose use is encouraged to achieve a specific purpose.

[0507] This invention is a system that efficiently supports plant cultivation and aims to realize user interaction that takes into account the user's emotions. The system combines plant image analysis and emotion recognition functions to provide the user with optimal cultivation advice.

[0508] Users take photos of plants using devices such as smartphones and tablets, and simultaneously input emotional data. This emotional data is provided via text or voice input. The device transmits this data to a cloud server via broadband communication. The transmitted data is typically packaged in JSON format using the HTTP protocol.

[0509] The server uses machine learning models for image analysis. This process employs software such as TensorFlow and OpenCV to identify the health status of plants based on the shape and color of their leaves. Furthermore, natural language processing techniques are used for sentiment analysis, utilizing libraries such as Python's NLTK library. This makes it possible to classify emotional tendencies based on user input into positive, negative, or neutral.

[0510] The server also generates cultivation advice based on the analysis results. Appropriate advice is created according to the analyzed state of the plant and the user's emotional tendencies. For example, if the plant is nutrient-deficient and the user is expressing anxiety, the server will provide detailed information on the type of fertilizer needed and how to use it, along with encouraging messages to alleviate the user's anxiety.

[0511] A concrete example of a prompt message would be, "I'm worried because my plant's leaves have recently turned yellow." Based on this input, the system analyzes the cause of the leaf discoloration and provides effective advice. By utilizing this system, users can cultivate their plants with a sense of emotional satisfaction.

[0512] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0513] Step 1:

[0514] The user takes a picture of a plant with their device and inputs sentiment data in text or voice. This process generates an image file of the plant and sentiment data as input data. The sentiment data is raw text or voice data based on the user's input.

[0515] Step 2:

[0516] The device sends the acquired image data and emotion data to a server in the cloud. The transmitted data is structured in JSON format and sent to the server over the network using the HTTP protocol. This process provides the server with the data necessary for analysis.

[0517] Step 3:

[0518] The server analyzes the received image data, using machine learning models to identify plant features. TensorFlow and OpenCV are used for image processing. The input data consists of plant images, and the output data is an identification result regarding the plant's health. For example, if the leaf color is abnormal, nutrient deficiency is identified as the cause.

[0519] Step 4:

[0520] The server uses natural language processing techniques to analyze sentiment data and determine the user's emotional tendencies. In this step, the input text is parsed using the Python NLTK library. The input is the user's sentiment data, and the output is a classification of positive, negative, or neutral sentiment.

[0521] Step 5:

[0522] The server generates appropriate advice based on the plant's health and the user's emotional state. This process creates text messages that take into account the plant's condition and the user's emotions. The output is the advice provided to the user.

[0523] Step 6:

[0524] The server sends the generated advice and recommended materials information back to the terminal. The data is transmitted again via the HTTP protocol over the network and is ready to be displayed on the terminal. In this step, the user receives clear advice and information on recommended materials as needed.

[0525] (Application Example 2)

[0526] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0527] In recent years, home gardening has become increasingly popular, but many users struggle to accurately assess the health of their plants and implement appropriate gardening methods. Furthermore, there is a lack of interaction that considers user emotions, and psychological support is rarely provided. Additionally, obtaining appropriate gardening advice and related supplies is difficult.

[0528] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0529] In this invention, the server includes means for performing mathematical procedures to analyze images of captured plants and identify their characteristics and condition; means for generating advice on how to cultivate the plants based on the identified characteristics and condition; and means for analyzing the user's emotions from voice and text input and adjusting the content and method of providing the advice based on the analysis results. This allows the user to receive appropriate cultivation advice along with psychological support, and to easily obtain the necessary items for cultivation.

[0530] "Analyzing images of photographed plants" means processing the photographic data of plants taken by users using computer vision technology to understand the characteristics and condition of those plants.

[0531] "Identifying characteristics and conditions" means identifying information such as the type of plant, its health, and its growth stage based on the analysis results.

[0532] "Performing mathematical procedures" refers to the process of applying various algorithms to captured images and performing data analysis.

[0533] "Generating advice on cultivation methods" means creating suggestions for optimal cultivation methods and treatments based on the plant's condition obtained through analysis.

[0534] "Analyzing user emotions from voice and text input" means using emotion recognition technology to identify emotions from the tone of voice spoken by the user and the text entered.

[0535] "Adjusting the content and method of providing advice" refers to adjusting the method of feedback and messaging to users based on the results of sentiment analysis.

[0536] "Supporting the acquisition of goods" means recommending materials and products necessary for plant cultivation and providing users with information on how to acquire them.

[0537] The embodiment of this invention begins with a user taking a picture of a plant with a smartphone or tablet and providing associated emotional data. The user's device transmits the captured image, along with the acquired audio and text data, to a server in the cloud. This server is implemented by integrating the following functions.

[0538] First, the server uses computer vision technology to analyze images and identify the characteristics and condition of the plants. Specifically, it commonly uses image analysis software frameworks such as OpenCV and TensorFlow. This identifies the type of plant and its health status.

[0539] Next, emotion recognition technology is used to analyze emotions from voice and text provided by the user. This analysis employs techniques such as speech tone analysis and Hugging Face Transformers as part of natural language processing (NLP). Based on the obtained emotion data, the server generates optimal parenting advice tailored to the user's psychological state.

[0540] Furthermore, the system recommends items corresponding to the generated training advice to the user and notifies them with information to help them acquire those items. For example, AWS Lambda or SageMaker can be used to have a cloud-based AI model provide specific training advice.

[0541] For example, if a user feels that "this plant doesn't seem healthy," the system uses sentiment analysis to understand the user's anxiety and then provides advice such as, "This plant needs a little more water. Please water it this much next time."

[0542] Example of a prompt:

[0543] "Provide emotional and care advice for plant maintenance for a worried user, especially focusing on water requirements and general care tips."

[0544] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0545] Step 1:

[0546] The user takes a picture of a plant with their smartphone and provides voice or text input. The input data consists of the plant image file and the user's voice or text data. The device collects this data and sends it to a cloud server when ready.

[0547] Step 2:

[0548] The server analyzes the received image data using computer vision software (e.g., OpenCV, TensorFlow). The input is image data of plants, and the output generates data that identifies the characteristics of the plants (e.g., species and health status). The server uses this analysis result to understand the condition of the plants.

[0549] Step 3:

[0550] The server analyzes the received audio or text data using an emotion recognition system (e.g., Hugging Face Transformers). The input is the user's audio file or text data, and the output identifies the user's emotional state (e.g., anxiety, confidence, etc.). The server uses this analysis to evaluate the user's psychological state.

[0551] Step 4:

[0552] The server generates cultivation advice using a generative AI model based on the analysis results of the plants and the user's emotional state. The inputs are plant condition data and user emotional data, and the output is specific advice to be provided to the user. The server adjusts the format of this advice according to the user's emotions.

[0553] Step 5:

[0554] Based on the generated advice, the server selects items recommended for the user and notifies the user, including this information. The input is the generated advice, and the output is notification data containing information about the recommended items and the advice. The server sends this data to the user's terminal for display.

[0555] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0556] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0557] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0558] [Fourth Embodiment]

[0559] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0560] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0561] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0562] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0563] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0564] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0565] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0566] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0567] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0568] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0569] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0570] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0571] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0572] This invention is a system that utilizes devices such as smartphones and tablets owned by plant growers to easily monitor the condition of plants and to support the selection of appropriate cultivation methods and materials.

[0573] The first step for the user is to take a picture of their plant using a smartphone or other device. The device sends this picture to a server in the cloud. The server then runs a dedicated algorithm based on the received image. This algorithm is pre-trained using machine learning techniques and can identify the type of plant, as well as the color, shape, and condition of its leaves and stems, through image analysis.

[0574] Based on the analysis results, the server identifies potential problems with the plant. For example, yellowing leaves may indicate a nitrogen deficiency, and unusual leaf shapes may suggest disease. Based on these analysis results, the server generates specific advice, including which nutrients are lacking and whether adjustments to sunlight or watering should be made.

[0575] Furthermore, the server provides information about the materials needed to solve the problem. This information includes details about the best materials for the plant type and the problem it is facing, such as specific fertilizers and soil amendments. The terminal notifies the user of this information and helps the user purchase the materials directly online.

[0576] As a concrete example, consider a scenario where a user takes a picture of their rose bush at home and sends it to the system. The server diagnoses that the rose is infected with powdery mildew and generates advice to use a fungicide effective against this disease. It also provides product information and online purchase links for the appropriate fungicide. This entire process allows the user to address plant problems quickly and effectively.

[0577] The following describes the processing flow.

[0578] Step 1:

[0579] Users take photos of plants using their smartphones or tablets. When taking photos, users pay attention to the angle and lighting so that the entire plant can be clearly seen.

[0580] Step 2:

[0581] The device receives the photographed plant image and adjusts the image size and resolution as needed. Next, the device sends the adjusted image data to a server in the cloud.

[0582] Step 3:

[0583] The server receives the image sent from the terminal. The server passes this image to an image analysis module, which uses a deep learning algorithm to analyze the characteristics and condition of the plants in the image. This analysis identifies the type of plant, leaf color and lesions, shape, and other characteristics.

[0584] Step 4:

[0585] The server evaluates the plant's health based on the analysis results. The evaluation process lists known problems (e.g., nutrient deficiencies, pest and disease effects) and determines whether those problems exist.

[0586] Step 5:

[0587] The server generates advice for the user based on the evaluation results. This advice includes things like watering, adjusting sunlight, and treating diseases.

[0588] Step 6:

[0589] The server lists recommended materials based on the plant type and condition. The list includes the specific name of the material, its effects, and a link to purchase it.

[0590] Step 7:

[0591] The device receives advice and recommended materials information sent from the server and notifies the user. The user can review the displayed information and purchase the recommended materials via the online store if necessary.

[0592] (Example 1)

[0593] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0594] One of the challenges faced by plant growers is the difficulty in accurately assessing plant health and quickly and effectively finding cultivation methods and problem-solving solutions. In particular, making expert judgments based on plant species and conditions, and providing cultivation advice and selecting materials accordingly, requires advanced knowledge and experience. Therefore, there is a need for methods that allow ordinary users to easily obtain this information and manage plant health.

[0595] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0596] In this invention, the server includes means for analyzing images of captured plants and executing an algorithm using machine learning techniques to identify the type of plant and the color, shape, and condition of its leaves and stems; means for automatically generating specific advice on how to cultivate and manage the plant based on the identified type, characteristics, and condition; and means including an interface that provides recommended material information based on the generated advice and assists in the online purchase of such materials. This makes it possible for users to easily understand the health of plants and quickly perform appropriate cultivation and problem solving without requiring specialized knowledge.

[0597] "Analyzing plant images" refers to processing plant image data transmitted from a terminal on a computer to identify the type of plant, as well as the color, shape, and condition of its leaves and stems.

[0598] An "algorithm using machine learning technology" is a technique for training computers to make accurate inferences based on vast amounts of data. In the context of plant image analysis, it refers to a program used to classify and recognize the features of image data based on a learned model.

[0599] "Specific advice on plant cultivation and management methods" refers to instructions and suggestions provided based on the analyzed state of the plant, including information on nutritional supplementation and environmental adjustments to maintain and improve plant health.

[0600] "Recommended materials information" refers to detailed information about products such as fertilizers, soil amendments, and fungicides that are recommended for use to address specific plant conditions or problems.

[0601] An "online purchase support interface" refers to a user interface that allows users to easily purchase recommended materials, and includes elements such as information provision and links during the purchase process.

[0602] This invention provides a system that allows plant growers to easily monitor the condition of their plants using devices such as smartphones and tablets. Users take photos of the plants they are growing with their devices and send them to a cloud server via a dedicated application. The devices used must have a camera function for taking photos and an internet connection.

[0603] The server executes an algorithm using machine learning techniques to analyze the received image data. This algorithm is built on platforms such as TensorFlow and PyTorch and can recognize the type and condition of plants. Based on the analysis results, it identifies the type of plant and any problems it has, and generates specific advice regarding appropriate cultivation methods and material selection.

[0604] Based on images taken by the user, the server can suggest nitrogen deficiency based on yellowing leaves and provide appropriate advice. The server then provides relevant material information based on the advice and notifies the user via their device. This notification feature helps the user check material details and purchase them online if necessary.

[0605] As a concrete example, consider a case where a user takes a picture of a rose they are growing at home and sends it to the system. The server analyzes the image and diagnoses that the rose has symptoms of powdery mildew. As a countermeasure, advice is generated to use a fungicide that is effective against this disease. Product information and purchase links for the appropriate fungicide are also provided, allowing the user to take action quickly.

[0606] An example of a prompt using a generative AI model is, "How can I find out about the white spots on the plant I'm growing?" In this way, users can easily obtain advice on specific problems.

[0607] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0608] Step 1:

[0609] The user takes a picture of the plant they are growing with their device. The input is image data of the plant. The device takes this image and prepares to send it to a server in the cloud via the application. In this process, the image data is compressed and converted into a format suitable for transfer. The output is image data ready for transmission. Specifically, the user opens the camera app, points the camera at the plant, and presses the shutter button.

[0610] Step 2:

[0611] The device sends captured image data to a server in the cloud. The input is image data from the user's device. The image data is transmitted over the internet using data communication technology. The server receives this data and stores it for subsequent processing. The output is the completion of storing the image data on the server. Specific operations include verifying that the device is connected to the internet and executing a program to send the data.

[0612] Step 3:

[0613] The server analyzes the received image data. The input is image data of plants stored on the server. It executes an algorithm using machine learning techniques to identify the type, characteristics, and condition of the plants in the image. During this process, data processing such as feature extraction and pattern matching is performed, and the identification result is obtained as output. Specifically, the process involves extracting necessary information from the image data via a machine learning model.

[0614] Step 4:

[0615] The server generates specific advice on plant cultivation and management based on the identification results. The input is the result of image recognition. In this process, appropriate cultivation information is retrieved from the database based on the identification data, and advice is created using a generative AI model. The output is an advice document to be provided to the user. The specific operations include executing database queries and generating text using a generative AI algorithm.

[0616] Step 5:

[0617] The server presents recommended material information based on the advice and prepares purchase support information. The input is the generated advice. It retrieves recommended material information from the database and generates links so that the user can easily purchase it. The output is material information and purchase links. The specific operation includes the extraction of relevant data and the link generation process.

[0618] Step 6:

[0619] The terminal notifies the user of generated advice and recommended materials. The input is the advice and materials provided by the server. The terminal displays this on the screen and prompts the user for confirmation. The output is information that the user can visually understand. Specific actions include generating notifications, updating the display, and providing information to the user.

[0620] (Application Example 1)

[0621] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0622] In plant cultivation, it is crucial to accurately understand the plant's condition and take prompt and precise action. However, traditional methods require users to assess the plant's condition themselves and find appropriate solutions, which presents challenges if they lack the necessary knowledge and experience. Furthermore, there is a need for efficient methods to constantly monitor plant health and provide timely care.

[0623] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0624] In this invention, the server includes means for analyzing images of captured plants and executing an algorithm to identify the characteristics and condition of the plants; means for generating advice on how to cultivate the plants based on the identified characteristics and condition; means for presenting recommended materials in accordance with the generated advice and assisting in the purchase of those materials; and an operative unit equipped with a camera for automatically monitoring the condition of the plants and diagnosing their health, and which autonomously provides advice based on the diagnostic results. This enables the user to understand the condition of the plants and efficiently and accurately select appropriate cultivation methods and materials.

[0625] An "algorithm" is a set of computational steps or processing methods designed to solve a specific problem.

[0626] "Characteristics" refer to the individual properties or attributes that an object possesses, and are key elements in identification and classification.

[0627] "State" refers to the physical and functional condition or circumstances of an object at a specific point in time.

[0628] "Advice" refers to information that provides guidance or instruction regarding actions or choices to be taken in a particular situation.

[0629] "Materials" refers to items or materials used for a specific purpose or use, and in this context, it refers to items necessary for plant cultivation.

[0630] A "photography device" is a device equipped with sensors that capture light and image information, and in this context, it is used to record the condition of plants.

[0631] An "operator" is a mechanism or device that performs a specific action or function, and in this context, it refers to a component that operates for the purpose of monitoring and diagnosing plants.

[0632] This invention is a system for understanding the condition of plants and providing appropriate advice and materials. The system consists of an operating unit equipped with a camera, a cloud server, and a user terminal.

[0633] The server receives images of plants acquired by the camera and executes an image analysis algorithm. This algorithm is trained on a generative AI model and identifies the characteristics and condition of the plants. Based on the identified results, the server generates specific advice on cultivation methods and suggests appropriate materials. This information is transmitted to the user's terminal via the cloud environment.

[0634] The device notifies the user of the advice and recommended materials received. Based on the displayed information, the user can purchase materials online and care for their plants. The system also allows for continuous monitoring of the plant's condition and timely responses to changes in its health.

[0635] As a concrete example, a robot periodically takes pictures of a plant at a user's home to monitor it and sends them to a server. If the server analyzes the images and detects any abnormalities in the plant's leaves, detailed advice on the cause and countermeasures is sent to the user's device. For example, prompts such as "Please diagnose the health of this plant" or "There are black spots on the plant. How should I deal with this?" are used to facilitate more specific analysis.

[0636] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0637] Step 1:

[0638] The user's terminal issues commands to the operating device, and the camera acquires images of plants. In this process, the camera photographs the plants at specific angles and distances, and adjusts the light intensity appropriately to obtain clear images. The input is the plant's actual visual information, and the output is the acquired digital image data.

[0639] Step 2:

[0640] The terminal transmits the acquired digital image data to a cloud server. Here, data is transmitted efficiently and securely via an internet connection, facilitating rapid processing on the server side. The input is digital image data, and the output is a signal indicating completion of the image data transmission to the server.

[0641] Step 3:

[0642] The server executes an image analysis algorithm using a generative AI model on the received image data. This algorithm identifies the characteristics and condition of plants in the image and extracts the necessary information. The input is the transmitted digital image data, and the output is analysis result data regarding the characteristics and condition of the plants.

[0643] Step 4:

[0644] The server generates advice on plant cultivation methods based on the analysis results. Here, prompts are provided to an AI model to suggest appropriate countermeasures and care methods based on the current state of the plants. The input is the analysis result data, and the output is advice on specific cultivation methods.

[0645] Step 5:

[0646] The server selects the most relevant material information related to the advice it generates and sends it to the user's terminal. This material information includes details of recommended products and purchase links. Inputs are the advice text and the material database, while output is material suggestion information for user notification.

[0647] Step 6:

[0648] The terminal notifies the user of received advice and material information. Based on this, the user can properly care for their plants and purchase materials online as needed. The input is advice and material information from the server, and the output is the information notification to the user.

[0649] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0650] This invention provides a system that enables user interaction while considering emotions in order to enhance plant cultivation support. This system combines plant image analysis and emotion recognition functions to provide optimal advice and feedback tailored to the user's needs.

[0651] Users take photos of plants with their smartphones or tablets and provide emotional data. This emotional data is estimated from the user's voice tone and text input. The device sends the acquired plant images and emotional data to a server in the cloud. The server analyzes the images using a dedicated algorithm to identify the plant's characteristics and condition. Based on this information, it generates advice on how to care for the plants.

[0652] The server uses an emotion engine to analyze the user's emotional tendencies. For example, if the user is feeling anxious, the server generates advice that includes encouraging messages and helpful guides. On the other hand, if the user is confident, it may provide more detailed technical information.

[0653] In user notifications, the way advice is presented is adjusted based on information obtained from sentiment analysis. For example, the UI tone and message style are changed according to the user's emotions, creating a more user-friendly and accessible interface. A specific example is providing step-by-step instructions and simple illustrated guides when a user is emotionally confused.

[0654] Furthermore, the server generates a list of recommended materials based on the analysis results and provides emotionally tailored purchasing support. For example, for users who are feeling anxious, it simultaneously presents additional information to persuade them of the necessity of the purchase, as well as past success stories. This system allows users to cultivate plants more effectively while gaining emotional satisfaction.

[0655] The following describes the processing flow.

[0656] Step 1:

[0657] Users take photos of plants with their smartphones or tablets and save them to their devices using a dedicated application. When taking photos, paying attention to the angle at which the entire plant is visible and the amount of light will facilitate accurate analysis.

[0658] Step 2:

[0659] The device collects emotional data from the user. This emotional data can be obtained by recording and analyzing the user's voice messages, or by using text input. The obtained data is passed to emotion recognition software to estimate the user's emotional state.

[0660] Step 3:

[0661] The device sends image data of plants it has photographed, along with estimated emotion data, to a server in the cloud. This transmission uses necessary security protocols to protect the data.

[0662] Step 4:

[0663] The server executes an image analysis algorithm to analyze the submitted plant images. In this process, the server identifies the plant species and health status, and extracts its characteristics. It also identifies disease conditions and environmental stresses that should be flagged.

[0664] Step 5:

[0665] The server uses an emotion engine to analyze the user's emotional data. For example, if the user is at ease, the server is configured to provide more detailed plant care information, and if they are anxious, it generates more supportive guidance.

[0666] Step 6:

[0667] The server generates advice based on the plant's analysis results and the user's emotional state. This advice includes specific plant care methods (nutrient supply, watering frequency, pest and disease control, etc.). It is also customized to include encouraging words and presentation methods appropriate to the user's situation.

[0668] Step 7:

[0669] The server generates a list of recommended materials and notifies the user. Based on sentiment data, it adds information to the user that emphasizes the effectiveness of the materials and facilitates the purchase process.

[0670] Step 8:

[0671] The device receives notifications from the server and presents them to the user in the most appropriate feedback format based on their emotional state. The user reviews the provided advice and recommended materials and purchases them online as needed. This allows the user to cultivate plants more effectively.

[0672] (Example 2)

[0673] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0674] Conventional plant cultivation support systems have struggled to provide effective advice that takes user emotions into consideration. Systems that only perform image analysis of plants lack feedback based on user emotions, and there was a need to improve user satisfaction with cultivation.

[0675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0676] In this invention, the server includes means for analyzing images of captured plants and performing information processing to identify the characteristics and state of the organism; means for generating instructions on how to manage the organism based on the identified characteristics and state; and means for analyzing user emotion data and adjusting the instructions based on that data. This makes it possible to provide accurate advice that takes the user's emotions into consideration.

[0677] "Information processing" is the technology used to manipulate data and generate results that meet specific purposes.

[0678] The term "living organism" refers to all living things, including plants, that have the characteristic of growing and changing in response to their environment.

[0679] An "input / output device" is a hardware and software system that enables the transmission and reception of data.

[0680] "Identification" is the act of finding specific characteristics or patterns and clarifying their type or state.

[0681] A "visualization device" is a means of displaying information to a user visually, and usually includes displays and the like.

[0682] "Instructions" are pieces of information or commands provided to request a specific action or response.

[0683] "Emotional data" refers to information that indicates the user's emotional state, and is data estimated from voice and text input.

[0684] A "network environment" is an infrastructure that enables multiple computers and devices to communicate information.

[0685] "Acquisition" refers to the act of obtaining necessary data or information.

[0686] "Recommended materials" are tools or materials whose use is encouraged to achieve a specific purpose.

[0687] This invention is a system that efficiently supports plant cultivation and aims to realize user interaction that takes into account the user's emotions. The system combines plant image analysis and emotion recognition functions to provide the user with optimal cultivation advice.

[0688] Users take photos of plants using devices such as smartphones and tablets, and simultaneously input emotional data. This emotional data is provided via text or voice input. The device transmits this data to a cloud server via broadband communication. The transmitted data is typically packaged in JSON format using the HTTP protocol.

[0689] The server uses machine learning models for image analysis. This process employs software such as TensorFlow and OpenCV to identify the health status of plants based on the shape and color of their leaves. Furthermore, natural language processing techniques are used for sentiment analysis, utilizing libraries such as Python's NLTK library. This makes it possible to classify emotional tendencies based on user input into positive, negative, or neutral.

[0690] The server also generates cultivation advice based on the analysis results. Appropriate advice is created according to the analyzed state of the plant and the user's emotional tendencies. For example, if the plant is nutrient-deficient and the user is expressing anxiety, the server will provide detailed information on the type of fertilizer needed and how to use it, along with encouraging messages to alleviate the user's anxiety.

[0691] A concrete example of a prompt message would be, "I'm worried because my plant's leaves have recently turned yellow." Based on this input, the system analyzes the cause of the leaf discoloration and provides effective advice. By utilizing this system, users can cultivate their plants with a sense of emotional satisfaction.

[0692] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0693] Step 1:

[0694] The user takes a picture of a plant with their device and inputs sentiment data in text or voice. This process generates an image file of the plant and sentiment data as input data. The sentiment data is raw text or voice data based on the user's input.

[0695] Step 2:

[0696] The device sends the acquired image data and emotion data to a server in the cloud. The transmitted data is structured in JSON format and sent to the server over the network using the HTTP protocol. This process provides the server with the data necessary for analysis.

[0697] Step 3:

[0698] The server analyzes the received image data, using machine learning models to identify plant features. TensorFlow and OpenCV are used for image processing. The input data consists of plant images, and the output data is an identification result regarding the plant's health. For example, if the leaf color is abnormal, nutrient deficiency is identified as the cause.

[0699] Step 4:

[0700] The server uses natural language processing techniques to analyze sentiment data and determine the user's emotional tendencies. In this step, the input text is parsed using the Python NLTK library. The input is the user's sentiment data, and the output is a classification of positive, negative, or neutral sentiment.

[0701] Step 5:

[0702] The server generates appropriate advice based on the plant's health and the user's emotional state. This process creates text messages that take into account the plant's condition and the user's emotions. The output is the advice provided to the user.

[0703] Step 6:

[0704] The server sends the generated advice and recommended materials information back to the terminal. The data is transmitted again via the HTTP protocol over the network and is ready to be displayed on the terminal. In this step, the user receives clear advice and information on recommended materials as needed.

[0705] (Application Example 2)

[0706] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0707] In recent years, home gardening has become increasingly popular, but many users struggle to accurately assess the health of their plants and implement appropriate gardening methods. Furthermore, there is a lack of interaction that considers user emotions, and psychological support is rarely provided. Additionally, obtaining appropriate gardening advice and related supplies is difficult.

[0708] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0709] In this invention, the server includes means for performing mathematical procedures to analyze images of captured plants and identify their characteristics and condition; means for generating advice on how to cultivate the plants based on the identified characteristics and condition; and means for analyzing the user's emotions from voice and text input and adjusting the content and method of providing the advice based on the analysis results. This allows the user to receive appropriate cultivation advice along with psychological support, and to easily obtain the necessary items for cultivation.

[0710] "Analyzing images of photographed plants" means processing the photographic data of plants taken by users using computer vision technology to understand the characteristics and condition of those plants.

[0711] "Identifying characteristics and conditions" means identifying information such as the type of plant, its health, and its growth stage based on the analysis results.

[0712] "Performing mathematical procedures" refers to the process of applying various algorithms to captured images and performing data analysis.

[0713] "Generating advice on cultivation methods" means creating suggestions for optimal cultivation methods and treatments based on the plant's condition obtained through analysis.

[0714] "Analyzing user emotions from voice and text input" means using emotion recognition technology to identify emotions from the tone of voice spoken by the user and the text entered.

[0715] "Adjusting the content and method of providing advice" refers to adjusting the method of feedback and messaging to users based on the results of sentiment analysis.

[0716] "Supporting the acquisition of goods" means recommending materials and products necessary for plant cultivation and providing users with information on how to acquire them.

[0717] The embodiment of this invention begins with a user taking a picture of a plant with a smartphone or tablet and providing associated emotional data. The user's device transmits the captured image, along with the acquired audio and text data, to a server in the cloud. This server is implemented by integrating the following functions.

[0718] First, the server uses computer vision technology to analyze images and identify the characteristics and condition of the plants. Specifically, it commonly uses image analysis software frameworks such as OpenCV and TensorFlow. This identifies the type of plant and its health status.

[0719] Next, emotion recognition technology is used to analyze emotions from voice and text provided by the user. This analysis employs techniques such as speech tone analysis and Hugging Face Transformers as part of natural language processing (NLP). Based on the obtained emotion data, the server generates optimal parenting advice tailored to the user's psychological state.

[0720] Furthermore, the system recommends items corresponding to the generated training advice to the user and notifies them with information to help them acquire those items. For example, AWS Lambda or SageMaker can be used to have a cloud-based AI model provide specific training advice.

[0721] For example, if a user feels that "this plant doesn't seem healthy," the system uses sentiment analysis to understand the user's anxiety and then provides advice such as, "This plant needs a little more water. Please water it this much next time."

[0722] Example of a prompt:

[0723] "Provide emotional and care advice for plant maintenance for a worried user, especially focusing on water requirements and general care tips."

[0724] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0725] Step 1:

[0726] The user takes a picture of a plant with their smartphone and provides voice or text input. The input data consists of the plant image file and the user's voice or text data. The device collects this data and sends it to a cloud server when ready.

[0727] Step 2:

[0728] The server analyzes the received image data using computer vision software (e.g., OpenCV, TensorFlow). The input is image data of plants, and the output generates data that identifies the characteristics of the plants (e.g., species and health status). The server uses this analysis result to understand the condition of the plants.

[0729] Step 3:

[0730] The server analyzes the received audio or text data using an emotion recognition system (e.g., Hugging Face Transformers). The input is the user's audio file or text data, and the output identifies the user's emotional state (e.g., anxiety, confidence, etc.). The server uses this analysis to evaluate the user's psychological state.

[0731] Step 4:

[0732] The server generates cultivation advice using a generative AI model based on the analysis results of the plants and the user's emotional state. The inputs are plant condition data and user emotional data, and the output is specific advice to be provided to the user. The server adjusts the format of this advice according to the user's emotions.

[0733] Step 5:

[0734] Based on the generated advice, the server selects items recommended for the user and notifies the user, including this information. The input is the generated advice, and the output is notification data containing information about the recommended items and the advice. The server sends this data to the user's terminal for display.

[0735] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0736] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0737] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0738] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0739] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0740] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0741] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0742] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0743] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0744] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0745] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0746] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0747] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0748] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0749] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0750] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0751] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0752] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0753] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0754] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0755] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0756] The following is further disclosed regarding the embodiments described above.

[0757] (Claim 1)

[0758] A means for analyzing images of plants taken and executing an algorithm to identify the characteristics and condition of those plants,

[0759] A means for generating advice on how to cultivate plants based on identified characteristics and conditions,

[0760] A means of providing recommended materials based on the generated advice and supporting the purchase of those materials,

[0761] A system that includes this.

[0762] (Claim 2)

[0763] The system according to claim 1, comprising an interface for acquiring images of plants and transmitting them to a cloud environment.

[0764] (Claim 3)

[0765] The system according to claim 1, further comprising display means for notifying the user of generated advice and recommended materials.

[0766] "Example 1"

[0767] (Claim 1)

[0768] A means for analyzing images of photographed plants and executing an algorithm using machine learning techniques to identify the type of plant, as well as the color, shape, and condition of its leaves and stems,

[0769] A means for automatically generating specific advice on how to cultivate and manage plants based on the identified species, characteristics, and condition,

[0770] A means including an interface that provides recommended material information based on the generated advice and assists in the online purchase of said materials,

[0771] A system that includes this.

[0772] (Claim 2)

[0773] The system according to claim 1, comprising a communication interface with a terminal for acquiring images of plants and transmitting all data to a cloud computing environment.

[0774] (Claim 3)

[0775] The system according to claim 1, further comprising output means for displaying generated advice and recommended materials on a user terminal and notifying the user.

[0776] "Application Example 1"

[0777] (Claim 1)

[0778] A means for analyzing images of plants taken and executing an algorithm to identify the characteristics and condition of those plants,

[0779] A means for generating advice on how to cultivate plants based on identified characteristics and conditions,

[0780] A means of providing recommended materials based on the generated advice and supporting the purchase of those materials,

[0781] An operating system equipped with a camera for automatically monitoring the condition of plants and diagnosing their health, comprising means for autonomously providing advice based on the diagnostic results,

[0782] A system that includes this.

[0783] (Claim 2)

[0784] The system according to claim 1, comprising an interface for acquiring images of plants and transmitting them to a cloud environment.

[0785] (Claim 3)

[0786] The system according to claim 1, further comprising display means for notifying the user of generated advice and recommended materials.

[0787] "Example 2 of combining an emotion engine"

[0788] (Claim 1)

[0789] A means for analyzing images of photographed plants and performing information processing to identify the characteristics and state of those organisms,

[0790] Means for generating instructions on how to manage an organism based on identified characteristics and conditions,

[0791] A means of analyzing user emotional data and adjusting instructions based on that data,

[0792] A means of presenting recommended materials in accordance with the generated instructions and supporting the acquisition of those materials,

[0793] A system that includes this.

[0794] (Claim 2)

[0795] The system according to claim 1, comprising an input / output device for acquiring images and emotion data of living organisms and transmitting them to a network environment.

[0796] (Claim 3)

[0797] The system according to claim 1, further comprising a visualization device for notifying the user of generated instructions and recommended materials.

[0798] "Application example 2 when combining with an emotional engine"

[0799] (Claim 1)

[0800] A means for analyzing images of photographed plants and performing mathematical procedures to identify the characteristics and condition of those plants,

[0801] A means for generating advice on how to cultivate plants based on identified characteristics and conditions,

[0802] A means of analyzing the user's emotions from voice and text input, and adjusting the content and method of providing advice based on the analysis results,

[0803] A means of presenting recommended items in accordance with the generated advice and assisting in the acquisition of those items,

[0804] A system that includes this.

[0805] (Claim 2)

[0806] The system according to claim 1, comprising a connection mechanism for acquiring plant images and emotion data and transmitting them to a digital environment.

[0807] (Claim 3)

[0808] The system according to claim 1, further comprising a display mechanism for notifying the user of generated advice and recommended items. [Explanation of Symbols]

[0809] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing images of plants taken and executing an algorithm to identify the characteristics and condition of those plants, A means for generating advice on how to cultivate plants based on identified characteristics and conditions, A means of providing recommended materials based on the generated advice and supporting the purchase of those materials, A system that includes this.

2. The system according to claim 1, comprising an interface for acquiring images of plants and transmitting them to a cloud environment.

3. The system according to claim 1, further comprising a display means for notifying the user of the generated advice and recommended materials.

Images