system

Abstract

We provide the system. [Solution] An information processing device provides an imaging means for photographing plant bodies, An identification means for identifying the type of plant by analyzing the captured image data, Information addition means for generating information related to the plant body based on the identified type, A risk assessment means for evaluating the instability of the plant using the aforementioned added information, A communication means for transmitting the aforementioned evaluation results to an external management organization, A system that includes this.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The risk caused by the falling of plants such as street trees has become a serious problem in urban areas and the like. In response to such problems, human resources and budgets are currently limited, making it difficult to quickly and efficiently evaluate risks and take appropriate management measures. Furthermore, there is no sufficient mechanism for citizens to actively participate and cooperate with management agencies to improve safety. Under such circumstances, there is a demand for a system that can easily identify the type of plant, evaluate its risk level, and provide information to management agencies.

Means for Solving the Problems

[0005] This invention includes an identification means for analyzing image data obtained by photographing a plant using an information processing device to identify its species. Based on the identified species, it includes an information addition means for generating information related to the plant, which includes information such as the physical distance of the plant and suggestions for maintenance methods. Furthermore, it includes a risk assessment means for evaluating the instability of the plant using the added information, and a communication means for transmitting this evaluation result to an external management organization. Thus, this invention realizes efficient management and improved safety of plants through cooperation between citizens and management organizations.

[0006] An "information processing device" is a device for inputting, analyzing, and outputting information, and is a computer system that performs calculations and data management.

[0007] "Imaging means" refers to a device or function for capturing an image of an object and collecting it as digital data.

[0008] "Plant body" refers to living organisms composed of organic matter that exist in nature, including street trees, and mainly refers to woody plants.

[0009] "Image data" refers to data that stores image information in a digital format and is represented as information that can be processed on a computer.

[0010] "Identification means" refers to a device or program that has the function of determining the type and characteristics of an object using a specific algorithm or process.

[0011] "Information addition means" refers to functions or processes for generating and adding new information to existing data.

[0012] A "risk assessment tool" is a function or device used to evaluate the risk of an object based on specific criteria and to clarify its level of risk.

[0013] "Communication means" refers to a mechanism or technology for transmitting data or information from one device to another.

[0014] A "managing body" is a public or private organization or group that is responsible for supervising, maintaining, and managing specific operations or facilities. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system that aims to improve the management and safety of plants, such as street trees, by having users photograph them using an information processing device. Specifically, users take images of plants using a device such as a smartphone or tablet. In addition to the captured image data, the device collects location information and user identification information, and transmits it to a cloud server as a data package.

[0037] The server typically passes the received image data to an AI, which uses image recognition technology to identify the type of plant. This process utilizes a pre-trained model to analyze the image and determine whether it is a cherry tree or a pine tree. Based on the obtained plant information, the generating AI creates relevant information such as recommended care methods and appropriate distances from roads. This information is overlaid as text on the image.

[0038] Next, the server uses a risk assessment tool to evaluate the risk of fallen trees from the image data containing the generated information. The risk level is classified into ranks such as "high," "medium," and "low," and countermeasures are proposed as needed.

[0039] The server transmits these evaluation results to the management agency via communication channels. The management agency receives this information and plans appropriate measures for the plants deemed dangerous. The pruning plan created by the AI ​​is then implemented after final approval by the management agency.

[0040] For example, if a user takes a picture of a cherry tree in their neighborhood, the server recognizes the image data and determines that it is a cherry tree. The generated information will include phrases such as "sufficient distance from the road" and "maintenance recommended." Based on this result, the plant's risk level is assessed as "medium" and sent to the management agency. The management agency can then use this information to plan and carry out pruning within a few months.

[0041] In this way, by utilizing information provided by citizens, management agencies can efficiently and quickly manage the plants.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] Users take pictures of street trees using their smartphones or tablets. The device automatically acquires location information and records user identification information when the image is taken.

[0045] Step 2:

[0046] The device creates a data package for sending the captured image data to the cloud server. This package includes image data, location information, and user identification information.

[0047] Step 3:

[0048] The device uses HTTPS to securely send data packages to the cloud server. The user is notified when the communication is successful.

[0049] Step 4:

[0050] The server typically passes the received image data to the AI. The AI ​​usually uses a pre-trained model to analyze the types of plants in the image and determine specific types such as "cherry blossoms" or "pine trees."

[0051] Step 5:

[0052] Based on the identified plant species, the server's AI generates relevant information such as care instructions and the appropriate distance from roads. This generated information is overlaid as text on the image.

[0053] Step 6:

[0054] The server uses a risk assessment tool to evaluate the risk of fallen trees based on the generated information. The evaluation is divided into categories such as "high," "medium," and "low," and necessary countermeasures are generated.

[0055] Step 7:

[0056] The server transmits data summarizing the evaluation results and countermeasures to an external management organization via communication means. The transmitted information is then processed by the management organization.

[0057] Step 8:

[0058] The management agency reviews the received information and develops a concrete action plan based on proposals, including pruning plans created by the AI ​​generator. The plan is finalized after considering on-site surveys and other factors.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] In modern society, managing and ensuring the safety of plants is a crucial issue in preserving the urban environment. However, conventional methods are not sufficiently efficient for gathering information and assessing the risk of each individual plant, potentially leading to delays in management and safety risks. Therefore, a new system is needed to quickly and accurately grasp the condition of plants and formulate appropriate management plans.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes imaging and data collection means for users to photograph plants, acquiring location information and identification information using a device, and storing them as a data package; image recognition means using a trained model to analyze the data package and determine the type of plant; and information generation means using generative AI to generate relevant advice information based on the determined type of plant. As a result, information acquired by users can be quickly analyzed, and management organizations can obtain advice to achieve appropriate plant management and improve safety.

[0064] "User" refers to an individual or group that uses the system to photograph plants and provide data.

[0065] "Plant body" refers to street trees and other natural plants that are subject to management, including their species and the management of such plants.

[0066] "Image acquisition and data collection means" refers to the function that allows the user to photograph a plant and collect location information and identification information along with the image data using the device.

[0067] A "data package" refers to a dataset that combines captured image data, location information, and identification information into a single file.

[0068] "Image recognition methods using pre-trained models" refer to methods that analyze image data using already trained models to identify plant species.

[0069] "Information generation method using AI generation" refers to a process that generates advice information regarding management and care based on the type of plant identified using AI technology.

[0070] A "risk assessment method" refers to a technique for evaluating the safety of plants based on generated advisory information and determining the degree of risk, such as the risk of trees falling.

[0071] "Data transmission means" refers to methods for communicating obtained evaluation results and generated information to external parties such as management organizations.

[0072] A "management organization" refers to an organization responsible for the management, conservation, and safety improvement of plant bodies.

[0073] This invention is a system that uses an information processing device for the purpose of improving the management and safety of plants. The system begins with a user taking a photograph of a plant using a device such as a smartphone or tablet. The device acquires the captured image data, along with location information such as GPS and user identification information, and transmits this data package to a cloud server.

[0074] When the server analyzes the received data package, it typically uses a deep learning-based AI model to identify images. Specifically, it utilizes a pre-trained model such as a convolutional neural network (CNN) to determine the type of plant. It might then produce a result such as, "This is a cherry blossom."

[0075] Next, the server uses a generative AI model to generate relevant advice for the identified plant species. During this process, prompts are input, and the generative AI model, for example, responds to prompts such as "Tell me how to care for a cherry tree," by suggesting care methods and preventative measures. The generated information is overlaid as text on the image. Image editing is performed using software such as OpenCV or PIL.

[0076] Furthermore, the server uses a risk assessment tool to evaluate the risk of trees falling based on the generated information. This evaluation is quantified into ranks such as "high," "medium," and "low," which helps management organizations develop countermeasures as needed.

[0077] These evaluation results are transmitted to the management organization via the server. Based on this information, the management organization can develop regular maintenance and pruning plans, enabling quick and efficient plant management.

[0078] As a concrete example, if a user takes a photo of a cherry tree in a nearby park, the device sends that information to a cloud server. The server identifies it as a "cherry tree" and generates advice such as "the distance from the road is appropriate" and "maintenance is recommended." Based on this information, the level of risk is assessed as "medium," sent to the management organization, and pruning is scheduled within a few months. This helps maintain a safe and healthy urban environment.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The user takes a picture of a plant using their device. This involves the user taking a picture of the plant with a camera app and acquiring image data. The input is an image of the plant, and the output is an image file saved on the device.

[0082] Step 2:

[0083] The device collects location and user identification information based on the captured image. It uses the device's GPS function to obtain location information and extracts identification information from user settings. The input is location and identification information acquired at the time of image capture, and the output is a data package containing this information.

[0084] Step 3:

[0085] The terminal sends the generated data package to the cloud server. The data package is uploaded to the server via an internet connection. The input is the data package, and the output is the data received and stored on the server.

[0086] Step 4:

[0087] The server analyzes the received data package and identifies the type of plant. It analyzes images using a standard AI model based on deep learning. The input is image data within the data package, and the output is the identified plant type (e.g., "cherry blossom").

[0088] Step 5:

[0089] The server uses a generative AI model to generate information based on plant species. It takes a specific prompt as input to the generative AI and generates relevant care instructions and advice. The input is the plant species and the prompt, and the output is the generated information text.

[0090] Step 6:

[0091] The server overlays the generated information onto the image. An image editing library is used to place the obtained text information onto the image. The input is the generated information text and the original image data, and the output is a new image with the information displayed.

[0092] Step 7:

[0093] The server uses the generated information and risk assessment tools to evaluate the risk level of the plant. It utilizes an algorithm to calculate a risk rank (e.g., "High," "Medium," "Low"). The input is an image with generated information, and the output is the risk rank as the evaluation result.

[0094] Step 8:

[0095] The server transmits the evaluation results to the management organization. It uses communication methods to inform the management organization of the evaluation results and related information. The input is the risk level rank and related information, and the output is the management organization's confirmation of receipt.

[0096] (Application Example 1)

[0097] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0098] In urban areas and public spaces, falling trees and poorly managed plants pose safety threats. Unstable plants, in particular, can cause disasters under severe weather conditions. However, monitoring and managing plants requires significant resources, and efficient monitoring and appropriate countermeasures are needed. Furthermore, a system is needed where residents of the community actively share information and work together to maintain safety.

[0099] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0100] In this invention, the server includes imaging means for photographing plant bodies using an information processing device, identification means for identifying the type of plant body by analyzing the captured image data, and information adding means for generating information related to the plant body based on the identified type. This enables local residents to assess the risk level of plant bodies, quickly provide information on unstable plant bodies to management agencies, and efficiently carry out safety management.

[0101] An "information processing device" is an electronic device that has the function of photographing plant bodies and analyzing the image data.

[0102] "Imaging means" refers to a component that includes a camera function for photographing plant bodies.

[0103] "Identification means" refers to an algorithm or process for analyzing captured image data to identify the type of plant.

[0104] The "information addition means" is a function that generates and adds information related to a plant based on the type of plant identified.

[0105] A "risk assessment tool" is a function that uses additional information to evaluate the instability of a plant and determine its level of risk.

[0106] "Communication means" refers to a function for transmitting evaluation results and related information to an external management organization.

[0107] An "input method" is an interface for users to provide information about plants in the public sphere.

[0108] A "generation method" is a function that generates recommended actions to contribute to the safety management of the local community based on the information provided.

[0109] To implement this invention, a smartphone or tablet device is first used as an information processing device. The user holds this device and photographs plants in a public area. The captured image is acquired in high resolution by the camera built into the device and temporarily stored in the device's storage. Metadata such as the user's location information and identification information is also added to the image data.

[0110] After taking a picture, the device sends this data to a cloud server. The server uses image recognition technologies such as TENSORFLOW® to analyze the received image data. This image recognition process identifies the type of plant, and based on the plant information identified by the "generative AI model," recommended care methods and information about the plant's instability are generated.

[0111] Furthermore, the server is equipped with a function to determine the level of risk. Using the generated information, the risk of trees falling is classified into numerical values ​​or ranks such as "high," "medium," and "low." These evaluation results are promptly transmitted to the management organization via cloud-based communication.

[0112] As a concrete example, suppose a citizen takes a picture of a cherry tree in a park with their smartphone and uploads the image to a cloud server via an application. The server analyzes the image data and recognizes it as a cherry tree. The AI ​​then generates a message such as "maintenance recommended," and the risk level is assessed as "medium," which is then reported to the management agency. Based on this information, the management agency plans pruning within a few months and takes action to maintain safety.

[0113] An example of a prompt message is as follows: "Location: 35.6895, 139.6917, Image: <Captured image data>, User ID: 123456, Please start analysis."

[0114] In this way, it can contribute to maintaining a safe and stable environment for the entire community.

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The user takes an image of a plant using a device. The input consists of the plant image, location information, and user identification information. The device acquires this input, adds metadata (location information and user ID) to the image data, and temporarily stores it.

[0118] Step 2:

[0119] The terminal sends temporarily stored image data and metadata to the cloud server. The output is the transmission of a data package to the server. Communication takes place to confirm that the data has been successfully uploaded to the server.

[0120] Step 3:

[0121] The server receives the image data and performs preprocessing. In this step, the data is prepared in a format suitable for applying an image recognition algorithm (e.g., a TensorFlow model). This process results in output image data that can identify plants.

[0122] Step 4:

[0123] The server uses image recognition technology to identify the type of plant from the received image. The input is pre-processed image data, and the output is information about the identified plant type. Based on this, information corresponding to a specific plant is selected.

[0124] Step 5:

[0125] The server uses a generative AI model to generate information about care instructions and instability based on the identified plant species. The input is plant species information, and the output is the generated related information. This generated information is overlaid on the plant image.

[0126] Step 6:

[0127] The server uses the generated information to assess the risk level of the plants. The input is the generated relevant information, and the output is a risk level ranking (high, medium, low). This assessment is determined based on the condition of the identified plants.

[0128] Step 7:

[0129] The server transmits the risk assessment results and related information to the management agency. The input consists of the risk assessment results and plant identification information, while the output is the transmission of information to the management agency. After transmission, the management agency can obtain the basic information necessary to plan countermeasures.

[0130] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0131] This invention is a plant management system that combines an emotion engine with an information processing device, and aims to create richer data by analyzing the emotions a user experiences when photographing plants. Specific embodiments for carrying out this invention will be described below.

[0132] Users take photos of street trees or specific plants using devices such as smartphones or tablets. The system also collects user emotional data during photography through speech and facial recognition cameras.

[0133] The device generates a data package containing plant image data, location information, user identification information, and user sentiment data, and sends it to a cloud server. The HTTPS protocol is used to ensure data security.

[0134] The server typically uses AI to analyze the received image data and identify the type of plant. For example, suppose a cherry tree is identified from the image. Based on this information, the generating AI creates additional information such as appropriate care methods and the appropriate distance from the road, and overlays it as an image with this information.

[0135] Next, the emotion engine analyzes the user's emotional data, and if positive emotions are detected, it highlights recommended information about the plant. Conversely, if negative emotions are recognized, it provides more detailed safety information and care advice.

[0136] The server uses these results to assess the risk level of the plants and sends the assessment results along with the sentiment analysis data to the management organization. This enables the development of more empathetic and adaptive plant management plans.

[0137] For example, if a user takes a picture of a cherry tree and says "beautiful," the emotion engine will detect a positive emotion. Based on this, the server will generate information highlighting the recommended time to view the cherry blossoms and simple maintenance tips. This information will be sent to the relevant authorities and can contribute to providing safety information to local residents and promoting tourism.

[0138] Thus, the system of the present invention enables advanced data provision and management by combining machine learning and emotion analysis in plant management.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] Users take photos of street trees or specific plants using their smartphones or tablets. During this process, the system captures images using the device's camera and also collects user emotion data using voice input and facial recognition features.

[0142] Step 2:

[0143] The device combines captured image data, location information, user identification information, and sentiment data into a data package. This data package is then prepared for transmission to a cloud server.

[0144] Step 3:

[0145] The device uses the HTTPS protocol to send data packages to the cloud server via secure communication. Once successful communication is confirmed, the user is notified.

[0146] Step 4:

[0147] The server typically passes the received image data to the AI, which then begins image analysis. This analysis uses a pre-trained model to identify the type of plant. For example, it might be identified as a cherry blossom.

[0148] Step 5:

[0149] Based on the identified plant species, the server uses a generative AI to generate relevant information such as maintenance methods and appropriate distances from roads, and integrates this information into the image as an overlay.

[0150] Step 6:

[0151] The server uses an emotion engine to analyze the emotional data provided by the user. When positive emotions are detected, a function is activated to highlight recommended information about the plant. On the other hand, if negative emotions are recognized, more detailed safety information is provided.

[0152] Step 7:

[0153] The server evaluates the risk level of the plant based on the generated information and the results of sentiment analysis. The risk level is clearly indicated by a ranking system of "high," "medium," and "low."

[0154] Step 8:

[0155] The server transmits the evaluation results and related information to an external management organization via communication means. The management organization then develops a plant management plan based on the received information.

[0156] Through this series of processes, we will build a system that provides information and plans to ensure the stability and safety of plants instantly, based on simple input from the user.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] Conventional plant management systems often identify plant species and provide basic care information, but they do not support personalized information delivery that takes user emotions into account. As a result, the emotional state in which users observe plants is not adequately reflected, sometimes leading to inappropriate plant management plans. Furthermore, the generation of information about plants is limited, resulting in a lack of data that can be used for plant management across an entire region. Therefore, there is a need for a system that utilizes user emotional data to enrich their experience while contributing to regional plant management.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes data acquisition means for photographing plants and collecting emotional data from the user's voice and facial expressions; identification means for analyzing the captured image data and identifying the type of plant; information generation means for generating information related to the plant using a generative artificial intelligence model based on the identified type and the user's emotional data; emotion analysis means for analyzing the emotional data, highlighting specific information when positive emotions are detected, and providing detailed safety information when negative emotions are detected; risk determination means for evaluating the instability of the plant using the generated information; and communication means for transmitting the evaluation results and emotional analysis results to an external management organization. This enables the provision of appropriate information according to the user's emotional state, and allows for more effective data analysis and planning for regional plant management.

[0162] The "data acquisition means" is a mechanism that collects emotional data through image data, the user's voice, and facial expressions when the user photographs a plant.

[0163] "Identification means" refers to a device that analyzes image data acquired by data acquisition means and has the function of identifying the type of plant.

[0164] The "information generation means" is a device for creating additional information related to a plant using a generative artificial intelligence model, based on the identified plant type and the user's emotional data.

[0165] An "emotion analysis tool" is a mechanism that analyzes emotional data obtained from users and adjusts the way information is displayed according to whether the emotions are positive or negative.

[0166] A "risk assessment method" is a means of evaluating the instability of a plant based on the generated information and determining its level of risk.

[0167] "Communication means" refers to the means of transmitting evaluation results and sentiment analysis data to an external management organization.

[0168] The system of this invention is designed to allow users to effectively manage plants using mobile devices such as smartphones and tablets. Users collect image data by taking pictures of street trees or specific plants. The device is equipped with a speech recognition function and a facial recognition camera, and by analyzing the user's voice and facial data, it acquires the user's emotional data.

[0169] The device integrates acquired image data, location information, user identification information, and sentiment data, and sends it to a cloud server as a data package. Data security is ensured by using the HTTPS protocol.

[0170] The server uses AI technology to analyze the received image data and identify the type of plant. This analysis utilizes a generative AI model to generate information on care methods and location based on the identified plant. Furthermore, an emotion engine analyzes the user's emotional data, highlighting information when positive emotions are recognized. Conversely, in the case of negative emotions, it provides detailed safety information and care advice.

[0171] For example, when a user takes a picture of a cherry tree and says "beautiful," the emotion engine detects the positive emotion, and the server generates information highlighting the best time to view the cherry blossoms and maintenance tips. This information is sent to the management organization and contributes to providing safety information to local residents and promoting tourism. An example of a prompt to the generating AI model would be, "Generate maintenance instructions from the photo of the cherry tree, and if the user's emotion is positive, highlight the recommended information."

[0172] This system allows users to obtain high-quality information tailored to their emotions, potentially adding new value to local plant management.

[0173] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0174] Step 1:

[0175] The user takes a picture of the plant using a device such as a smartphone or tablet. The input is an image of the plant. The device uses a speech and facial recognition camera to collect emotional data from the user's voice and facial expressions. The output of this step is image data and emotional data. Specifically, the user uses the device's camera to take a picture of the plant, and at the same time, emotional input is provided through voice and facial expressions.

[0176] Step 2:

[0177] The device integrates captured image data, location information, user identification information, and sentiment data to generate a data package. The various data acquired in Step 1 are used as input. Data processing involves formatting this information into a single data package. The output is a data package transmitted via HTTPS. Specifically, the device organizes the data and prepares it for transmission to the cloud server via a communication module.

[0178] Step 3:

[0179] The server receives a data package sent from the terminal. The input includes the received image data. The server uses AI technology to analyze the image data and identify the type of plant. The data processing involves extracting features from the plant image and comparing them with an existing database to identify the type. The output is information about the identified plant type. Specifically, the AI ​​engine processes the image and obtains the identification result.

[0180] Step 4:

[0181] The server uses a generative AI model to generate information related to the plant based on the identified plant type and the user's emotional data. The inputs used are the plant type information and emotional data obtained in step 3. As data processing, this information is used to generate care instructions and recommendations. The output is the generated additional information. Specifically, the AI ​​model generates data such as care know-how and viewing times.

[0182] Step 5:

[0183] The server analyzes emotional data, highlighting specific information when positive emotions are detected and providing detailed safety information when negative emotions are detected. The input is the user's emotional data, which is then analyzed. Data processing involves customizing the information based on the intensity of the emotions. The output is customized plant information. Specifically, the system adjusts how the generated information is displayed.

[0184] Step 6:

[0185] Ultimately, the server transmits all analysis results and sentiment analysis data to the management agency for use in plant management across the region. The information and analysis results obtained in step 5 are used as input. A communication module is used to transmit data to the management agency. The output is the agency's receipt and use of the information. Specifically, the information is integrated into the administrator's interface to aid in the development of regional plans.

[0186] (Application Example 2)

[0187] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0188] Modern plant management systems require accurate identification of plant species and conditions, but a problem is that they fail to consider the crucial information of user emotions. Furthermore, there is a lack of methods to provide individually customized information and suggestions about plants to enrich the customer experience in retail stores. Therefore, there is a need for information delivery methods that respond to user emotions.

[0189] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0190] In this invention, the server includes, via an information processing device, an imaging means for capturing a plant; an identification means for identifying the type of plant by analyzing the captured image data; an information generation means for generating information related to the plant based on the identified type; a risk assessment means for evaluating the instability of the plant using the generated information; an emotion analysis means for collecting and analyzing emotion data from the user; a recommendation means for providing specific suggestions for additional information or items related to the plant based on the user's emotion data; and a communication means for transmitting the evaluation results and additional information to an external control organization. This enables comprehensive management of plants and the provision of customized information.

[0191] An "information processing device" is an electronic device used to manipulate and analyze data, and is a device that has the function of processing image data of plants and user emotion data.

[0192] "Imaging means" refers to a mechanism for visually capturing plant bodies, and is an image acquisition device that includes cameras and sensors.

[0193] "Identification means" refers to a device or program that has the function of analyzing captured data to identify the type of plant.

[0194] "Information generation means" refers to a device or program that has the function of creating relevant information based on the type of plant identified and providing it to the user.

[0195] A "risk assessment tool" is a device or program that uses generated information to analyze and evaluate the instability of the plant's condition and health.

[0196] "Emotional analysis means" refers to a device or program for analyzing emotional data collected from users and understanding its content.

[0197] A "recommendation tool" is a device or program that has the function of suggesting specific information or products to a user based on the user's emotional data and plant information.

[0198] "Communication means" refers to a device or program that has the function of transmitting evaluation results or additional information to an external control body or other system.

[0199] To implement this invention, it is necessary to construct a system that combines an information processing device, an emotion analysis engine, and a generative AI model. The user takes pictures of plants in a physical store using a terminal such as a smartphone or tablet. The terminal is equipped with a camera module and an audio input module, which are used to acquire image data of the plants and emotion data from the user's speech.

[0200] The terminal packages the collected data and sends it to the cloud server. To ensure security, the data is communicated using the HTTPS protocol. On the server, the image data is first analyzed using image recognition software (e.g., Google® Cloud Vision API) to identify the type of plant. Based on this information, an information generation system creates information on how to cultivate the plant and provides recommendations.

[0201] Next, the server analyzes the user's emotional data using emotion analysis software (e.g., Microsoft® Azure® Emotion API). If the emotion is positive, the server uses a generative AI model to provide recommendations to the user. This information includes cultivation advice tailored to the plant's condition and appropriate related products.

[0202] For example, if a user takes a picture of a rose in the store and says "beautiful," the sentiment analysis system will analyze the positive emotion, and the recommendation system will suggest fertilizers and pruning tools suitable for growing roses. This entire sequence of information is also transmitted to an external control organization, contributing to store inventory management and improving customer satisfaction.

[0203] Example of a prompt:

[0204] "A user took a photo of a rose and commented, 'Beautiful.' Please generate a positive purchase suggestion regarding rose cultivation."

[0205] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0206] Step 1:

[0207] The user uses the device to photograph a plant and says "beautiful." Image and audio data are collected as input. The device packages this data and sends it to a cloud server. The HTTPS protocol is used for transmission, ensuring data security.

[0208] Step 2:

[0209] The server analyzes the plant using image recognition software (e.g., Google Cloud Vision API) based on the received image data. In this step, the image data is used as input for analysis, and information about the plant's species is obtained as output. Specific actions are then taken to identify the plant as a rose.

[0210] Step 3:

[0211] The server uses the received audio data to analyze the user's speech using emotion analysis software (e.g., Microsoft Azure Emotion API). In this step, the audio data is used as input for analysis, and the user's emotion data is obtained as output. Based on the analysis results, specific actions are taken to detect the user's positive emotions.

[0212] Step 4:

[0213] The server integrates plant species information and user sentiment data, and uses a generative AI model to generate recommendations for the user. Based on this input data, it generates advice and related product information as output. Specifically, it provides optimal methods for growing roses and recommends suitable products.

[0214] Step 5:

[0215] The server sends the generated recommendation information to the user's device. The transmitted information is displayed on the screen of the smartphone or tablet. Based on this information, the user can take specific actions such as considering how to care for plants or purchasing related products.

[0216] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0217] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0218] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0219] [Second Embodiment]

[0220] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0221] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0222] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0223] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0224] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0225] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0226] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0227] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0228] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0229] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0230] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0231] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0232] This invention is a system that aims to improve the management and safety of plants, such as street trees, by having users photograph them using an information processing device. Specifically, users take images of plants using a device such as a smartphone or tablet. In addition to the captured image data, the device collects location information and user identification information, and transmits it to a cloud server as a data package.

[0233] The server typically passes the received image data to an AI, which uses image recognition technology to identify the type of plant. This process utilizes a pre-trained model to analyze the image and determine whether it is a cherry tree or a pine tree. Based on the obtained plant information, the generating AI creates relevant information such as recommended care methods and appropriate distances from roads. This information is overlaid as text on the image.

[0234] Next, the server uses a risk assessment tool to evaluate the risk of fallen trees from the image data containing the generated information. The risk level is classified into ranks such as "high," "medium," and "low," and countermeasures are proposed as needed.

[0235] The server transmits these evaluation results to the management agency via communication channels. The management agency receives this information and plans appropriate measures for the plants deemed dangerous. The pruning plan created by the AI ​​is then implemented after final approval by the management agency.

[0236] For example, if a user takes a picture of a cherry tree in their neighborhood, the server recognizes the image data and determines that it is a cherry tree. The generated information will include phrases such as "sufficient distance from the road" and "maintenance recommended." Based on this result, the plant's risk level is assessed as "medium" and sent to the management agency. The management agency can then use this information to plan and carry out pruning within a few months.

[0237] In this way, by utilizing information provided by citizens, management agencies can efficiently and quickly manage the plants.

[0238] The following describes the processing flow.

[0239] Step 1:

[0240] Users take pictures of street trees using their smartphones or tablets. The device automatically acquires location information and records user identification information when the image is taken.

[0241] Step 2:

[0242] The device creates a data package for sending the captured image data to the cloud server. This package includes image data, location information, and user identification information.

[0243] Step 3:

[0244] The device uses HTTPS to securely send data packages to the cloud server. The user is notified when the communication is successful.

[0245] Step 4:

[0246] The server typically passes the received image data to the AI. The AI ​​usually uses a pre-trained model to analyze the types of plants in the image and determine specific types such as "cherry blossoms" or "pine trees."

[0247] Step 5:

[0248] Based on the identified plant species, the server's AI generates relevant information such as care instructions and the appropriate distance from roads. This generated information is overlaid as text on the image.

[0249] Step 6:

[0250] The server uses a risk assessment tool to evaluate the risk of fallen trees based on the generated information. The evaluation is divided into categories such as "high," "medium," and "low," and necessary countermeasures are generated.

[0251] Step 7:

[0252] The server transmits data summarizing the evaluation results and countermeasures to an external management organization via communication means. The transmitted information is then processed by the management organization.

[0253] Step 8:

[0254] The management agency reviews the received information and develops a concrete action plan based on proposals, including pruning plans created by the AI ​​generator. The plan is finalized after considering on-site surveys and other factors.

[0255] (Example 1)

[0256] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0257] In modern society, managing and ensuring the safety of plants is a crucial issue in preserving the urban environment. However, conventional methods are not sufficiently efficient for gathering information and assessing the risk of each individual plant, potentially leading to delays in management and safety risks. Therefore, a new system is needed to quickly and accurately grasp the condition of plants and formulate appropriate management plans.

[0258] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0259] In this invention, the server includes imaging and data collection means for users to photograph plants, acquiring location information and identification information using a device, and storing them as a data package; image recognition means using a trained model to analyze the data package and determine the type of plant; and information generation means using generative AI to generate relevant advice information based on the determined type of plant. As a result, information acquired by users can be quickly analyzed, and management organizations can obtain advice to achieve appropriate plant management and improve safety.

[0260] "User" refers to an individual or group that uses the system to photograph plants and provide data.

[0261] "Plant body" refers to street trees and other natural plants that are subject to management, including their species and the management of such plants.

[0262] "Image acquisition and data collection means" refers to the function that allows the user to photograph a plant and collect location information and identification information along with the image data using the device.

[0263] A "data package" refers to a dataset that combines captured image data, location information, and identification information into a single file.

[0264] "Image recognition methods using pre-trained models" refer to methods that analyze image data using already trained models to identify plant species.

[0265] "Information generation method using AI generation" refers to a process that generates advice information regarding management and care based on the type of plant identified using AI technology.

[0266] A "risk assessment method" refers to a technique for evaluating the safety of plants based on generated advisory information and determining the degree of risk, such as the risk of trees falling.

[0267] "Data transmission means" refers to methods for communicating obtained evaluation results and generated information to external parties such as management organizations.

[0268] A "management organization" refers to an organization responsible for the management, conservation, and safety improvement of plant bodies.

[0269] This invention is a system that uses an information processing device for the purpose of improving the management and safety of plants. The system begins with a user taking a photograph of a plant using a device such as a smartphone or tablet. The device acquires the captured image data, along with location information such as GPS and user identification information, and transmits this data package to a cloud server.

[0270] When the server analyzes the received data package, it typically uses a deep learning-based AI model to identify images. Specifically, it utilizes a pre-trained model such as a convolutional neural network (CNN) to determine the type of plant. It might then produce a result such as, "This is a cherry blossom."

[0271] Next, the server uses a generative AI model to generate relevant advice for the identified plant species. During this process, prompts are input, and the generative AI model, for example, responds to prompts such as "Tell me how to care for a cherry tree," by suggesting care methods and preventative measures. The generated information is overlaid as text on the image. Image editing is performed using software such as OpenCV or PIL.

[0272] Furthermore, the server uses a risk assessment tool to evaluate the risk of trees falling based on the generated information. This evaluation is quantified into ranks such as "high," "medium," and "low," which helps management organizations develop countermeasures as needed.

[0273] These evaluation results are transmitted to the management organization via the server. Based on this information, the management organization can develop regular maintenance and pruning plans, enabling quick and efficient plant management.

[0274] As a concrete example, if a user takes a photo of a cherry tree in a nearby park, the device sends that information to a cloud server. The server identifies it as a "cherry tree" and generates advice such as "the distance from the road is appropriate" and "maintenance is recommended." Based on this information, the level of risk is assessed as "medium," sent to the management organization, and pruning is scheduled within a few months. This helps maintain a safe and healthy urban environment.

[0275] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0276] Step 1:

[0277] The user takes a picture of a plant using a terminal. This includes the action of the user taking a picture of the plant with a camera app and obtaining image data. The input is an image of the plant, and the output is an image file saved in the terminal.

[0278] Step 2:

[0279] Based on the captured image, the terminal collects location information and user identification information. It obtains location information using the terminal's GPS function and extracts identification information from user settings. The input is the location information and identification information obtained at the timing of image capture, and the output is a data package containing this information.

[0280] Step 3:

[0281] The terminal sends the generated data package to the cloud server. It uploads the data package to the server through an Internet connection. The input is the data package, and the output is the data received and saved in the server.

[0282] Step 4:

[0283] The server analyzes the received data package and determines the type of the plant. It analyzes the image using a normal AI model based on deep learning. The input is the image data within the data package, and the output is the determined type of the plant (e.g., "cherry blossom").

[0284] Step 5:

[0285] The server uses a generation AI model to generate information based on the type of the plant. It passes a specific prompt sentence as input to the generation AI and generates relevant cultivation methods and helpful information. The input is the type of the plant and the prompt sentence, and the output is the generated information text.

[0286] Step 6:

[0287] The server overlays the generated information onto the image. An image editing library is used to place the obtained text information onto the image. The input is the generated information text and the original image data, and the output is a new image with the information displayed.

[0288] Step 7:

[0289] The server uses the generated information and risk assessment tools to evaluate the risk level of the plant. It utilizes an algorithm to calculate a risk rank (e.g., "High," "Medium," "Low"). The input is an image with generated information, and the output is the risk rank as the evaluation result.

[0290] Step 8:

[0291] The server transmits the evaluation results to the management organization. It uses communication methods to inform the management organization of the evaluation results and related information. The input is the risk level rank and related information, and the output is the management organization's confirmation of receipt.

[0292] (Application Example 1)

[0293] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0294] In urban areas and public spaces, falling trees and poorly managed plants pose safety threats. Unstable plants, in particular, can cause disasters under severe weather conditions. However, monitoring and managing plants requires significant resources, and efficient monitoring and appropriate countermeasures are needed. Furthermore, a system is needed where residents of the community actively share information and work together to maintain safety.

[0295] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0296] In this invention, the server includes imaging means for photographing plant bodies using an information processing device, identification means for identifying the type of plant body by analyzing the captured image data, and information adding means for generating information related to the plant body based on the identified type. This enables local residents to assess the risk level of plant bodies, quickly provide information on unstable plant bodies to management agencies, and efficiently carry out safety management.

[0297] An "information processing device" is an electronic device that has the function of photographing plant bodies and analyzing the image data.

[0298] "Imaging means" refers to a component that includes a camera function for photographing plant bodies.

[0299] "Identification means" refers to an algorithm or process for analyzing captured image data to identify the type of plant.

[0300] The "information addition means" is a function that generates and adds information related to a plant based on the type of plant identified.

[0301] A "risk assessment tool" is a function that uses additional information to evaluate the instability of a plant and determine its level of risk.

[0302] "Communication means" refers to a function for transmitting evaluation results and related information to an external management organization.

[0303] An "input method" is an interface for users to provide information about plants in the public sphere.

[0304] A "generation method" is a function that generates recommended actions to contribute to the safety management of the local community based on the information provided.

[0305] To implement this invention, first, a smartphone or a tablet terminal as an information processing device is used. The user holds this terminal and takes a picture of a plant body existing in a public area. The captured image is acquired at a high resolution by a camera built into the terminal and temporarily stored in the terminal's storage. Metadata such as the user's location information and identification information is also added to the image data.

[0306] After shooting, the terminal transmits this data to a cloud server. The server utilizes image recognition technologies such as TensorFlow to analyze the received image data. Through this image recognition process, the type of the plant body is identified, and based on the plant information identified by the "generative AI model", information on recommended maintenance methods and the instability of the plant body is generated.

[0307] Furthermore, the server is equipped with a function to determine the degree of danger. Using the generated information, the risk of the plant body falling is classified in terms of numerical evaluation or ranks such as "high", "medium", and "low". These evaluation results are promptly transmitted to the management agency through communication means on the cloud.

[0308] As a specific example, suppose a citizen takes a picture of a cherry tree in a park with a smartphone and uploads the image to a cloud server through an application. The server analyzes the image data and recognizes it as a cherry tree. Also, the generative AI generates a message such as "Recommended maintenance", and the risk assessment is notified to the management agency as "medium". Based on this information, the management agency plans pruning within a few months and takes actions to maintain safety.

[0309] Examples of prompt sentences are as follows: "Location information: 35.6895, 139.6917, Image: <Captured image data>, User ID: 123456, Please start analysis."

[0310] In this way, it can contribute to maintaining a safe and stable environment for the entire local community.

[0311] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0312] Step 1:

[0313] The user takes an image of a plant using a device. The input consists of the plant image, location information, and user identification information. The device acquires this input, adds metadata (location information and user ID) to the image data, and temporarily stores it.

[0314] Step 2:

[0315] The terminal sends temporarily stored image data and metadata to the cloud server. The output is the transmission of a data package to the server. Communication takes place to confirm that the data has been successfully uploaded to the server.

[0316] Step 3:

[0317] The server receives the image data and performs preprocessing. In this step, the data is prepared in a format suitable for applying an image recognition algorithm (e.g., a TensorFlow model). This process results in output image data that can identify plants.

[0318] Step 4:

[0319] The server uses image recognition technology to identify the type of plant from the received image. The input is pre-processed image data, and the output is information about the identified plant type. Based on this, information corresponding to a specific plant is selected.

[0320] Step 5:

[0321] The server uses a generative AI model to generate information about care instructions and instability based on the identified plant species. The input is plant species information, and the output is the generated related information. This generated information is overlaid on the plant image.

[0322] Step 6:

[0323] The server uses the generated information to assess the risk level of the plants. The input is the generated relevant information, and the output is a risk level ranking (high, medium, low). This assessment is determined based on the condition of the identified plants.

[0324] Step 7:

[0325] The server transmits the risk assessment results and related information to the management agency. The input consists of the risk assessment results and plant identification information, while the output is the transmission of information to the management agency. After transmission, the management agency can obtain the basic information necessary to plan countermeasures.

[0326] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0327] This invention is a plant management system that combines an emotion engine with an information processing device, and aims to create richer data by analyzing the emotions a user experiences when photographing plants. Specific embodiments for carrying out this invention will be described below.

[0328] Users take photos of street trees or specific plants using devices such as smartphones or tablets. The system also collects user emotional data during photography through speech and facial recognition cameras.

[0329] The device generates a data package containing plant image data, location information, user identification information, and user sentiment data, and sends it to a cloud server. The HTTPS protocol is used to ensure data security.

[0330] The server typically uses AI to analyze the received image data and identify the type of plant. For example, suppose a cherry tree is identified from the image. Based on this information, the generating AI creates additional information such as appropriate care methods and the appropriate distance from the road, and overlays it as an image with this information.

[0331] Next, the emotion engine analyzes the user's emotional data, and if positive emotions are detected, it highlights recommended information about the plant. Conversely, if negative emotions are recognized, it provides more detailed safety information and care advice.

[0332] The server uses these results to assess the risk level of the plants and sends the assessment results along with the sentiment analysis data to the management organization. This enables the development of more empathetic and adaptive plant management plans.

[0333] For example, if a user takes a picture of a cherry tree and says "beautiful," the emotion engine will detect a positive emotion. Based on this, the server will generate information highlighting the recommended time to view the cherry blossoms and simple maintenance tips. This information will be sent to the relevant authorities and can contribute to providing safety information to local residents and promoting tourism.

[0334] Thus, the system of the present invention enables advanced data provision and management by combining machine learning and emotion analysis in plant management.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] Users take photos of street trees or specific plants using their smartphones or tablets. During this process, the system captures images using the device's camera and also collects user emotion data using voice input and facial recognition features.

[0338] Step 2:

[0339] The device combines captured image data, location information, user identification information, and sentiment data into a data package. This data package is then prepared for transmission to a cloud server.

[0340] Step 3:

[0341] The device uses the HTTPS protocol to send data packages to the cloud server via secure communication. Once successful communication is confirmed, the user is notified.

[0342] Step 4:

[0343] The server typically passes the received image data to the AI, which then begins image analysis. This analysis uses a pre-trained model to identify the type of plant. For example, it might be identified as a cherry blossom.

[0344] Step 5:

[0345] Based on the identified plant species, the server uses a generative AI to generate relevant information such as maintenance methods and appropriate distances from roads, and integrates this information into the image as an overlay.

[0346] Step 6:

[0347] The server uses an emotion engine to analyze the emotional data provided by the user. When positive emotions are detected, a function is activated to highlight recommended information about the plant. On the other hand, if negative emotions are recognized, more detailed safety information is provided.

[0348] Step 7:

[0349] The server evaluates the risk level of the plant based on the generated information and the results of sentiment analysis. The risk level is clearly indicated by a ranking system of "high," "medium," and "low."

[0350] Step 8:

[0351] The server transmits the evaluation results and related information to an external management organization via communication means. The management organization then develops a plant management plan based on the received information.

[0352] Through this series of processes, we will build a system that provides information and plans to ensure the stability and safety of plants instantly, based on simple input from the user.

[0353] (Example 2)

[0354] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0355] Conventional plant management systems often identify plant species and provide basic care information, but they do not support personalized information delivery that takes user emotions into account. As a result, the emotional state in which users observe plants is not adequately reflected, sometimes leading to inappropriate plant management plans. Furthermore, the generation of information about plants is limited, resulting in a lack of data that can be used for plant management across an entire region. Therefore, there is a need for a system that utilizes user emotional data to enrich their experience while contributing to regional plant management.

[0356] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0357] In this invention, the server includes data acquisition means for photographing plants and collecting emotional data from the user's voice and facial expressions; identification means for analyzing the captured image data and identifying the type of plant; information generation means for generating information related to the plant using a generative artificial intelligence model based on the identified type and the user's emotional data; emotion analysis means for analyzing the emotional data, highlighting specific information when positive emotions are detected, and providing detailed safety information when negative emotions are detected; risk determination means for evaluating the instability of the plant using the generated information; and communication means for transmitting the evaluation results and emotional analysis results to an external management organization. This enables the provision of appropriate information according to the user's emotional state, and allows for more effective data analysis and planning for regional plant management.

[0358] The "data acquisition means" is a mechanism that collects emotional data through image data, the user's voice, and facial expressions when the user photographs a plant.

[0359] "Identification means" refers to a device that analyzes image data acquired by data acquisition means and has the function of identifying the type of plant.

[0360] The "information generation means" is a device for creating additional information related to a plant using a generative artificial intelligence model, based on the identified plant type and the user's emotional data.

[0361] An "emotion analysis tool" is a mechanism that analyzes emotional data obtained from users and adjusts the way information is displayed according to whether the emotions are positive or negative.

[0362] A "risk assessment method" is a means of evaluating the instability of a plant based on the generated information and determining its level of risk.

[0363] "Communication means" refers to the means of transmitting evaluation results and sentiment analysis data to an external management organization.

[0364] The system of this invention is designed to allow users to effectively manage plants using mobile devices such as smartphones and tablets. Users collect image data by taking pictures of street trees or specific plants. The device is equipped with a speech recognition function and a facial recognition camera, and by analyzing the user's voice and facial data, it acquires the user's emotional data.

[0365] The device integrates acquired image data, location information, user identification information, and sentiment data, and sends it to a cloud server as a data package. Data security is ensured by using the HTTPS protocol.

[0366] The server uses AI technology to analyze the received image data and identify the type of plant. This analysis utilizes a generative AI model to generate information on care methods and location based on the identified plant. Furthermore, an emotion engine analyzes the user's emotional data, highlighting information when positive emotions are recognized. Conversely, in the case of negative emotions, it provides detailed safety information and care advice.

[0367] For example, when a user takes a picture of a cherry tree and says "beautiful," the emotion engine detects the positive emotion, and the server generates information highlighting the best time to view the cherry blossoms and maintenance tips. This information is sent to the management organization and contributes to providing safety information to local residents and promoting tourism. An example of a prompt to the generating AI model would be, "Generate maintenance instructions from the photo of the cherry tree, and if the user's emotion is positive, highlight the recommended information."

[0368] This system allows users to obtain high-quality information tailored to their emotions, potentially adding new value to local plant management.

[0369] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0370] Step 1:

[0371] The user takes a picture of the plant using a device such as a smartphone or tablet. The input is an image of the plant. The device uses a speech and facial recognition camera to collect emotional data from the user's voice and facial expressions. The output of this step is image data and emotional data. Specifically, the user uses the device's camera to take a picture of the plant, and at the same time, emotional input is provided through voice and facial expressions.

[0372] Step 2:

[0373] The device integrates captured image data, location information, user identification information, and sentiment data to generate a data package. The various data acquired in Step 1 are used as input. Data processing involves formatting this information into a single data package. The output is a data package transmitted via HTTPS. Specifically, the device organizes the data and prepares it for transmission to the cloud server via a communication module.

[0374] Step 3:

[0375] The server receives a data package sent from the terminal. The input includes the received image data. The server uses AI technology to analyze the image data and identify the type of plant. The data processing involves extracting features from the plant image and comparing them with an existing database to identify the type. The output is information about the identified plant type. Specifically, the AI ​​engine processes the image and obtains the identification result.

[0376] Step 4:

[0377] The server uses a generative AI model to generate information related to the plant based on the identified plant type and the user's emotional data. The inputs used are the plant type information and emotional data obtained in step 3. As data processing, this information is used to generate care instructions and recommendations. The output is the generated additional information. Specifically, the AI ​​model generates data such as care know-how and viewing times.

[0378] Step 5:

[0379] The server analyzes emotional data, highlighting specific information when positive emotions are detected and providing detailed safety information when negative emotions are detected. The input is the user's emotional data, which is then analyzed. Data processing involves customizing the information based on the intensity of the emotions. The output is customized plant information. Specifically, the system adjusts how the generated information is displayed.

[0380] Step 6:

[0381] Ultimately, the server transmits all analysis results and sentiment analysis data to the management agency for use in plant management across the region. The information and analysis results obtained in step 5 are used as input. A communication module is used to transmit data to the management agency. The output is the agency's receipt and use of the information. Specifically, the information is integrated into the administrator's interface to aid in the development of regional plans.

[0382] (Application Example 2)

[0383] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0384] Modern plant management systems require accurate identification of plant species and conditions, but a problem is that they fail to consider the crucial information of user emotions. Furthermore, there is a lack of methods to provide individually customized information and suggestions about plants to enrich the customer experience in retail stores. Therefore, there is a need for information delivery methods that respond to user emotions.

[0385] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0386] In this invention, the server includes, via an information processing device, an imaging means for capturing a plant; an identification means for identifying the type of plant by analyzing the captured image data; an information generation means for generating information related to the plant based on the identified type; a risk assessment means for evaluating the instability of the plant using the generated information; an emotion analysis means for collecting and analyzing emotion data from the user; a recommendation means for providing specific suggestions for additional information or items related to the plant based on the user's emotion data; and a communication means for transmitting the evaluation results and additional information to an external control organization. This enables comprehensive management of plants and the provision of customized information.

[0387] An "information processing device" is an electronic device used to manipulate and analyze data, and is a device that has the function of processing image data of plants and user emotion data.

[0388] "Imaging means" refers to a mechanism for visually capturing plant bodies, and is an image acquisition device that includes cameras and sensors.

[0389] "Identification means" refers to a device or program that has the function of analyzing captured data to identify the type of plant.

[0390] "Information generation means" refers to a device or program that has the function of creating relevant information based on the type of plant identified and providing it to the user.

[0391] A "risk assessment tool" is a device or program that uses generated information to analyze and evaluate the instability of the plant's condition and health.

[0392] "Emotional analysis means" refers to a device or program for analyzing emotional data collected from users and understanding its content.

[0393] A "recommendation tool" is a device or program that has the function of suggesting specific information or products to a user based on the user's emotional data and plant information.

[0394] "Communication means" refers to a device or program that has the function of transmitting evaluation results or additional information to an external control body or other system.

[0395] To implement this invention, it is necessary to construct a system that combines an information processing device, an emotion analysis engine, and a generative AI model. The user takes pictures of plants in a physical store using a terminal such as a smartphone or tablet. The terminal is equipped with a camera module and an audio input module, which are used to acquire image data of the plants and emotion data from the user's speech.

[0396] The terminal packages the collected data and sends it to the cloud server. To ensure security, the data is communicated using the HTTPS protocol. On the server, the image data is first analyzed using image recognition software (e.g., Google Cloud Vision API) to identify the type of plant. Based on this information, an information generation system creates information on how to cultivate the plant and provides recommendations.

[0397] Next, the server analyzes the user's emotional data using emotion analysis software (e.g., Microsoft Azure Emotion API). If the emotion is positive, the server uses a generative AI model to provide recommendations to the user. This information includes cultivation advice tailored to the plant's condition and appropriate related products.

[0398] For example, if a user takes a picture of a rose in the store and says "beautiful," the sentiment analysis system will analyze the positive emotion, and the recommendation system will suggest fertilizers and pruning tools suitable for growing roses. This entire sequence of information is also transmitted to an external control organization, contributing to store inventory management and improving customer satisfaction.

[0399] Example of a prompt:

[0400] "A user took a photo of a rose and commented, 'Beautiful.' Please generate a positive purchase suggestion regarding rose cultivation."

[0401] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0402] Step 1:

[0403] The user uses the device to photograph a plant and says "beautiful." Image and audio data are collected as input. The device packages this data and sends it to a cloud server. The HTTPS protocol is used for transmission, ensuring data security.

[0404] Step 2:

[0405] The server analyzes the plant using image recognition software (e.g., Google Cloud Vision API) based on the received image data. In this step, the image data is used as input for analysis, and information about the plant's species is obtained as output. Specific actions are then taken to identify the plant as a rose.

[0406] Step 3:

[0407] The server uses the received audio data to analyze the user's speech using emotion analysis software (e.g., Microsoft Azure Emotion API). In this step, the audio data is used as input for analysis, and the user's emotion data is obtained as output. Based on the analysis results, specific actions are taken to detect the user's positive emotions.

[0408] Step 4:

[0409] The server integrates plant species information and user sentiment data, and uses a generative AI model to generate recommendations for the user. Based on this input data, it generates advice and related product information as output. Specifically, it provides optimal methods for growing roses and recommends suitable products.

[0410] Step 5:

[0411] The server sends the generated recommendation information to the user's device. The transmitted information is displayed on the screen of the smartphone or tablet. Based on this information, the user can take specific actions such as considering how to care for plants or purchasing related products.

[0412] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0413] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0414] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0415] [Third Embodiment]

[0416] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0417] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0418] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0419] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0420] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0421] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0422] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0423] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0424] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0425] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0426] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0427] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0428] This invention is a system that aims to improve the management and safety of plants, such as street trees, by having users photograph them using an information processing device. Specifically, users take images of plants using a device such as a smartphone or tablet. In addition to the captured image data, the device collects location information and user identification information, and transmits it to a cloud server as a data package.

[0429] The server typically passes the received image data to an AI, which uses image recognition technology to identify the type of plant. This process utilizes a pre-trained model to analyze the image and determine whether it is a cherry tree or a pine tree. Based on the obtained plant information, the generating AI creates relevant information such as recommended care methods and appropriate distances from roads. This information is overlaid as text on the image.

[0430] Next, the server uses a risk assessment tool to evaluate the risk of fallen trees from the image data containing the generated information. The risk level is classified into ranks such as "high," "medium," and "low," and countermeasures are proposed as needed.

[0431] The server transmits these evaluation results to the management agency via communication channels. The management agency receives this information and plans appropriate measures for the plants deemed dangerous. The pruning plan created by the AI ​​is then implemented after final approval by the management agency.

[0432] For example, if a user takes a picture of a cherry tree in their neighborhood, the server recognizes the image data and determines that it is a cherry tree. The generated information will include phrases such as "sufficient distance from the road" and "maintenance recommended." Based on this result, the plant's risk level is assessed as "medium" and sent to the management agency. The management agency can then use this information to plan and carry out pruning within a few months.

[0433] In this way, by utilizing information provided by citizens, management agencies can efficiently and quickly manage the plants.

[0434] The following describes the processing flow.

[0435] Step 1:

[0436] Users take pictures of street trees using their smartphones or tablets. The device automatically acquires location information and records user identification information when the image is taken.

[0437] Step 2:

[0438] The device creates a data package for sending the captured image data to the cloud server. This package includes image data, location information, and user identification information.

[0439] Step 3:

[0440] The device uses HTTPS to securely send data packages to the cloud server. The user is notified when the communication is successful.

[0441] Step 4:

[0442] The server typically passes the received image data to the AI. The AI ​​usually uses a pre-trained model to analyze the types of plants in the image and determine specific types such as "cherry blossoms" or "pine trees."

[0443] Step 5:

[0444] Based on the identified plant species, the server's AI generates relevant information such as care instructions and the appropriate distance from roads. This generated information is overlaid as text on the image.

[0445] Step 6:

[0446] The server uses a risk assessment tool to evaluate the risk of fallen trees based on the generated information. The evaluation is divided into categories such as "high," "medium," and "low," and necessary countermeasures are generated.

[0447] Step 7:

[0448] The server transmits data summarizing the evaluation results and countermeasures to an external management organization via communication means. The transmitted information is then processed by the management organization.

[0449] Step 8:

[0450] The management agency reviews the received information and develops a concrete action plan based on proposals, including pruning plans created by the AI ​​generator. The plan is finalized after considering on-site surveys and other factors.

[0451] (Example 1)

[0452] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0453] In modern society, managing and ensuring the safety of plants is a crucial issue in preserving the urban environment. However, conventional methods are not sufficiently efficient for gathering information and assessing the risk of each individual plant, potentially leading to delays in management and safety risks. Therefore, a new system is needed to quickly and accurately grasp the condition of plants and formulate appropriate management plans.

[0454] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0455] In this invention, the server includes imaging and data collection means for users to photograph plants, acquiring location information and identification information using a device, and storing them as a data package; image recognition means using a trained model to analyze the data package and determine the type of plant; and information generation means using generative AI to generate relevant advice information based on the determined type of plant. As a result, information acquired by users can be quickly analyzed, and management organizations can obtain advice to achieve appropriate plant management and improve safety.

[0456] "User" refers to an individual or group that uses the system to photograph plants and provide data.

[0457] "Plant body" refers to street trees and other natural plants that are subject to management, including their species and the management of such plants.

[0458] "Image acquisition and data collection means" refers to the function that allows the user to photograph a plant and collect location information and identification information along with the image data using the device.

[0459] A "data package" refers to a dataset that combines captured image data, location information, and identification information into a single file.

[0460] "Image recognition methods using pre-trained models" refer to methods that analyze image data using already trained models to identify plant species.

[0461] "Information generation method using AI generation" refers to a process that generates advice information regarding management and care based on the type of plant identified using AI technology.

[0462] A "risk assessment method" refers to a technique for evaluating the safety of plants based on generated advisory information and determining the degree of risk, such as the risk of trees falling.

[0463] "Data transmission means" refers to methods for communicating obtained evaluation results and generated information to external parties such as management organizations.

[0464] A "management organization" refers to an organization responsible for the management, conservation, and safety improvement of plant bodies.

[0465] This invention is a system that uses an information processing device for the purpose of improving the management and safety of plants. The system begins with a user taking a photograph of a plant using a device such as a smartphone or tablet. The device acquires the captured image data, along with location information such as GPS and user identification information, and transmits this data package to a cloud server.

[0466] When the server analyzes the received data package, it typically uses a deep learning-based AI model to identify images. Specifically, it utilizes a pre-trained model such as a convolutional neural network (CNN) to determine the type of plant. It might then produce a result such as, "This is a cherry blossom."

[0467] Next, the server uses a generative AI model to generate relevant advice for the identified plant species. During this process, prompts are input, and the generative AI model, for example, responds to prompts such as "Tell me how to care for a cherry tree," by suggesting care methods and preventative measures. The generated information is overlaid as text on the image. Image editing is performed using software such as OpenCV or PIL.

[0468] Furthermore, the server uses a risk assessment tool to evaluate the risk of trees falling based on the generated information. This evaluation is quantified into ranks such as "high," "medium," and "low," which helps management organizations develop countermeasures as needed.

[0469] These evaluation results are transmitted to the management organization via the server. Based on this information, the management organization can develop regular maintenance and pruning plans, enabling quick and efficient plant management.

[0470] As a concrete example, if a user takes a photo of a cherry tree in a nearby park, the device sends that information to a cloud server. The server identifies it as a "cherry tree" and generates advice such as "the distance from the road is appropriate" and "maintenance is recommended." Based on this information, the level of risk is assessed as "medium," sent to the management organization, and pruning is scheduled within a few months. This helps maintain a safe and healthy urban environment.

[0471] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0472] Step 1:

[0473] The user takes a picture of a plant using their device. This involves the user taking a picture of the plant with a camera app and acquiring image data. The input is an image of the plant, and the output is an image file saved on the device.

[0474] Step 2:

[0475] The device collects location and user identification information based on the captured image. It uses the device's GPS function to obtain location information and extracts identification information from user settings. The input is location and identification information acquired at the time of image capture, and the output is a data package containing this information.

[0476] Step 3:

[0477] The terminal sends the generated data package to the cloud server. The data package is uploaded to the server via an internet connection. The input is the data package, and the output is the data received and stored on the server.

[0478] Step 4:

[0479] The server analyzes the received data package and identifies the type of plant. It analyzes images using a standard AI model based on deep learning. The input is image data within the data package, and the output is the identified plant type (e.g., "cherry blossom").

[0480] Step 5:

[0481] The server uses a generative AI model to generate information based on plant species. It takes a specific prompt as input to the generative AI and generates relevant care instructions and advice. The input is the plant species and the prompt, and the output is the generated information text.

[0482] Step 6:

[0483] The server overlays the generated information onto the image. An image editing library is used to place the obtained text information onto the image. The input is the generated information text and the original image data, and the output is a new image with the information displayed.

[0484] Step 7:

[0485] The server uses the generated information and risk assessment tools to evaluate the risk level of the plant. It utilizes an algorithm to calculate a risk rank (e.g., "High," "Medium," "Low"). The input is an image with generated information, and the output is the risk rank as the evaluation result.

[0486] Step 8:

[0487] The server transmits the evaluation results to the management organization. It uses communication methods to inform the management organization of the evaluation results and related information. The input is the risk level rank and related information, and the output is the management organization's confirmation of receipt.

[0488] (Application Example 1)

[0489] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0490] In urban areas and public spaces, falling trees and poorly managed plants pose safety threats. Unstable plants, in particular, can cause disasters under severe weather conditions. However, monitoring and managing plants requires significant resources, and efficient monitoring and appropriate countermeasures are needed. Furthermore, a system is needed where residents of the community actively share information and work together to maintain safety.

[0491] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0492] In this invention, the server includes imaging means for photographing plant bodies using an information processing device, identification means for identifying the type of plant body by analyzing the captured image data, and information adding means for generating information related to the plant body based on the identified type. This enables local residents to assess the risk level of plant bodies, quickly provide information on unstable plant bodies to management agencies, and efficiently carry out safety management.

[0493] An "information processing device" is an electronic device that has the function of photographing plant bodies and analyzing the image data.

[0494] "Imaging means" refers to a component that includes a camera function for photographing plant bodies.

[0495] "Identification means" refers to an algorithm or process for analyzing captured image data to identify the type of plant.

[0496] The "information addition means" is a function that generates and adds information related to a plant based on the type of plant identified.

[0497] A "risk assessment tool" is a function that uses additional information to evaluate the instability of a plant and determine its level of risk.

[0498] "Communication means" refers to a function for transmitting evaluation results and related information to an external management organization.

[0499] An "input method" is an interface for users to provide information about plants in the public sphere.

[0500] A "generation method" is a function that generates recommended actions to contribute to the safety management of the local community based on the information provided.

[0501] To implement this invention, a smartphone or tablet device is first used as an information processing device. The user holds this device and photographs plants in a public area. The captured image is acquired in high resolution by the camera built into the device and temporarily stored in the device's storage. Metadata such as the user's location information and identification information is also added to the image data.

[0502] After taking a picture, the device sends this data to a cloud server. The server uses image recognition technologies such as TensorFlow to analyze the received image data. This image recognition process identifies the type of plant, and based on the plant information identified by the "generative AI model," recommended care methods and information about the plant's instability are generated.

[0503] Furthermore, the server is equipped with a function to determine the level of risk. Using the generated information, the risk of trees falling is classified into numerical values ​​or ranks such as "high," "medium," and "low." These evaluation results are promptly transmitted to the management organization via cloud-based communication.

[0504] As a concrete example, suppose a citizen takes a picture of a cherry tree in a park with their smartphone and uploads the image to a cloud server via an application. The server analyzes the image data and recognizes it as a cherry tree. The AI ​​then generates a message such as "maintenance recommended," and the risk level is assessed as "medium," which is then reported to the management agency. Based on this information, the management agency plans pruning within a few months and takes action to maintain safety.

[0505] An example of a prompt message is as follows: "Location: 35.6895, 139.6917, Image: <Captured image data>, User ID: 123456, Please start analysis."

[0506] In this way, it can contribute to maintaining a safe and stable environment for the entire community.

[0507] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0508] Step 1:

[0509] The user takes an image of a plant using a device. The input consists of the plant image, location information, and user identification information. The device acquires this input, adds metadata (location information and user ID) to the image data, and temporarily stores it.

[0510] Step 2:

[0511] The terminal sends temporarily stored image data and metadata to the cloud server. The output is the transmission of a data package to the server. Communication takes place to confirm that the data has been successfully uploaded to the server.

[0512] Step 3:

[0513] The server receives the image data and performs preprocessing. In this step, the data is prepared in a format suitable for applying an image recognition algorithm (e.g., a TensorFlow model). This process results in output image data that can identify plants.

[0514] Step 4:

[0515] The server uses image recognition technology to identify the type of plant from the received image. The input is pre-processed image data, and the output is information about the identified plant type. Based on this, information corresponding to a specific plant is selected.

[0516] Step 5:

[0517] The server uses a generative AI model to generate information about care instructions and instability based on the identified plant species. The input is plant species information, and the output is the generated related information. This generated information is overlaid on the plant image.

[0518] Step 6:

[0519] The server uses the generated information to assess the risk level of the plants. The input is the generated relevant information, and the output is a risk level ranking (high, medium, low). This assessment is determined based on the condition of the identified plants.

[0520] Step 7:

[0521] The server transmits the risk assessment results and related information to the management agency. The input consists of the risk assessment results and plant identification information, while the output is the transmission of information to the management agency. After transmission, the management agency can obtain the basic information necessary to plan countermeasures.

[0522] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0523] This invention is a plant management system that combines an emotion engine with an information processing device, and aims to create richer data by analyzing the emotions a user experiences when photographing plants. Specific embodiments for carrying out this invention will be described below.

[0524] Users take photos of street trees or specific plants using devices such as smartphones or tablets. The system also collects user emotional data during photography through speech and facial recognition cameras.

[0525] The device generates a data package containing plant image data, location information, user identification information, and user sentiment data, and sends it to a cloud server. The HTTPS protocol is used to ensure data security.

[0526] The server typically uses AI to analyze the received image data and identify the type of plant. For example, suppose a cherry tree is identified from the image. Based on this information, the generating AI creates additional information such as appropriate care methods and the appropriate distance from the road, and overlays it as an image with this information.

[0527] Next, the emotion engine analyzes the user's emotional data, and if positive emotions are detected, it highlights recommended information about the plant. Conversely, if negative emotions are recognized, it provides more detailed safety information and care advice.

[0528] The server uses these results to assess the risk level of the plants and sends the assessment results along with the sentiment analysis data to the management organization. This enables the development of more empathetic and adaptive plant management plans.

[0529] For example, if a user takes a picture of a cherry tree and says "beautiful," the emotion engine will detect a positive emotion. Based on this, the server will generate information highlighting the recommended time to view the cherry blossoms and simple maintenance tips. This information will be sent to the relevant authorities and can contribute to providing safety information to local residents and promoting tourism.

[0530] Thus, the system of the present invention enables advanced data provision and management by combining machine learning and emotion analysis in plant management.

[0531] The following describes the processing flow.

[0532] Step 1:

[0533] Users take photos of street trees or specific plants using their smartphones or tablets. During this process, the system captures images using the device's camera and also collects user emotion data using voice input and facial recognition features.

[0534] Step 2:

[0535] The device combines captured image data, location information, user identification information, and sentiment data into a data package. This data package is then prepared for transmission to a cloud server.

[0536] Step 3:

[0537] The device uses the HTTPS protocol to send data packages to the cloud server via secure communication. Once successful communication is confirmed, the user is notified.

[0538] Step 4:

[0539] The server typically passes the received image data to the AI, which then begins image analysis. This analysis uses a pre-trained model to identify the type of plant. For example, it might be identified as a cherry blossom.

[0540] Step 5:

[0541] Based on the identified plant species, the server uses a generative AI to generate relevant information such as maintenance methods and appropriate distances from roads, and integrates this information into the image as an overlay.

[0542] Step 6:

[0543] The server uses an emotion engine to analyze the emotional data provided by the user. When positive emotions are detected, a function is activated to highlight recommended information about the plant. On the other hand, if negative emotions are recognized, more detailed safety information is provided.

[0544] Step 7:

[0545] The server evaluates the risk level of the plant based on the generated information and the results of sentiment analysis. The risk level is clearly indicated by a ranking system of "high," "medium," and "low."

[0546] Step 8:

[0547] The server transmits the evaluation results and related information to an external management organization via communication means. The management organization then develops a plant management plan based on the received information.

[0548] Through this series of processes, we will build a system that provides information and plans to ensure the stability and safety of plants instantly, based on simple input from the user.

[0549] (Example 2)

[0550] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0551] Conventional plant management systems often identify plant species and provide basic care information, but they do not support personalized information delivery that takes user emotions into account. As a result, the emotional state in which users observe plants is not adequately reflected, sometimes leading to inappropriate plant management plans. Furthermore, the generation of information about plants is limited, resulting in a lack of data that can be used for plant management across an entire region. Therefore, there is a need for a system that utilizes user emotional data to enrich their experience while contributing to regional plant management.

[0552] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0553] In this invention, the server includes data acquisition means for photographing plants and collecting emotional data from the user's voice and facial expressions; identification means for analyzing the captured image data and identifying the type of plant; information generation means for generating information related to the plant using a generative artificial intelligence model based on the identified type and the user's emotional data; emotion analysis means for analyzing the emotional data, highlighting specific information when positive emotions are detected, and providing detailed safety information when negative emotions are detected; risk determination means for evaluating the instability of the plant using the generated information; and communication means for transmitting the evaluation results and emotional analysis results to an external management organization. This enables the provision of appropriate information according to the user's emotional state, and allows for more effective data analysis and planning for regional plant management.

[0554] The "data acquisition means" is a mechanism that collects emotional data through image data, the user's voice, and facial expressions when the user photographs a plant.

[0555] "Identification means" refers to a device that analyzes image data acquired by data acquisition means and has the function of identifying the type of plant.

[0556] The "information generation means" is a device for creating additional information related to a plant using a generative artificial intelligence model, based on the identified plant type and the user's emotional data.

[0557] An "emotion analysis tool" is a mechanism that analyzes emotional data obtained from users and adjusts the way information is displayed according to whether the emotions are positive or negative.

[0558] A "risk assessment method" is a means of evaluating the instability of a plant based on the generated information and determining its level of risk.

[0559] "Communication means" refers to the means of transmitting evaluation results and sentiment analysis data to an external management organization.

[0560] The system of this invention is designed to allow users to effectively manage plants using mobile devices such as smartphones and tablets. Users collect image data by taking pictures of street trees or specific plants. The device is equipped with a speech recognition function and a facial recognition camera, and by analyzing the user's voice and facial data, it acquires the user's emotional data.

[0561] The device integrates acquired image data, location information, user identification information, and sentiment data, and sends it to a cloud server as a data package. Data security is ensured by using the HTTPS protocol.

[0562] The server uses AI technology to analyze the received image data and identify the type of plant. This analysis utilizes a generative AI model to generate information on care methods and location based on the identified plant. Furthermore, an emotion engine analyzes the user's emotional data, highlighting information when positive emotions are recognized. Conversely, in the case of negative emotions, it provides detailed safety information and care advice.

[0563] For example, when a user takes a picture of a cherry tree and says "beautiful," the emotion engine detects the positive emotion, and the server generates information highlighting the best time to view the cherry blossoms and maintenance tips. This information is sent to the management organization and contributes to providing safety information to local residents and promoting tourism. An example of a prompt to the generating AI model would be, "Generate maintenance instructions from the photo of the cherry tree, and if the user's emotion is positive, highlight the recommended information."

[0564] This system allows users to obtain high-quality information tailored to their emotions, potentially adding new value to local plant management.

[0565] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0566] Step 1:

[0567] The user takes a picture of the plant using a device such as a smartphone or tablet. The input is an image of the plant. The device uses a speech and facial recognition camera to collect emotional data from the user's voice and facial expressions. The output of this step is image data and emotional data. Specifically, the user uses the device's camera to take a picture of the plant, and at the same time, emotional input is provided through voice and facial expressions.

[0568] Step 2:

[0569] The device integrates captured image data, location information, user identification information, and sentiment data to generate a data package. The various data acquired in Step 1 are used as input. Data processing involves formatting this information into a single data package. The output is a data package transmitted via HTTPS. Specifically, the device organizes the data and prepares it for transmission to the cloud server via a communication module.

[0570] Step 3:

[0571] The server receives a data package sent from the terminal. The input includes the received image data. The server uses AI technology to analyze the image data and identify the type of plant. The data processing involves extracting features from the plant image and comparing them with an existing database to identify the type. The output is information about the identified plant type. Specifically, the AI ​​engine processes the image and obtains the identification result.

[0572] Step 4:

[0573] The server uses a generative AI model to generate information related to the plant based on the identified plant type and the user's emotional data. The inputs used are the plant type information and emotional data obtained in step 3. As data processing, this information is used to generate care instructions and recommendations. The output is the generated additional information. Specifically, the AI ​​model generates data such as care know-how and viewing times.

[0574] Step 5:

[0575] The server analyzes emotional data, highlighting specific information when positive emotions are detected and providing detailed safety information when negative emotions are detected. The input is the user's emotional data, which is then analyzed. Data processing involves customizing the information based on the intensity of the emotions. The output is customized plant information. Specifically, the system adjusts how the generated information is displayed.

[0576] Step 6:

[0577] Ultimately, the server transmits all analysis results and sentiment analysis data to the management agency for use in plant management across the region. The information and analysis results obtained in step 5 are used as input. A communication module is used to transmit data to the management agency. The output is the agency's receipt and use of the information. Specifically, the information is integrated into the administrator's interface to aid in the development of regional plans.

[0578] (Application Example 2)

[0579] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0580] Modern plant management systems require accurate identification of plant species and conditions, but a problem is that they fail to consider the crucial information of user emotions. Furthermore, there is a lack of methods to provide individually customized information and suggestions about plants to enrich the customer experience in retail stores. Therefore, there is a need for information delivery methods that respond to user emotions.

[0581] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0582] In this invention, the server includes, via an information processing device, an imaging means for capturing a plant; an identification means for identifying the type of plant by analyzing the captured image data; an information generation means for generating information related to the plant based on the identified type; a risk assessment means for evaluating the instability of the plant using the generated information; an emotion analysis means for collecting and analyzing emotion data from the user; a recommendation means for providing specific suggestions for additional information or items related to the plant based on the user's emotion data; and a communication means for transmitting the evaluation results and additional information to an external control organization. This enables comprehensive management of plants and the provision of customized information.

[0583] An "information processing device" is an electronic device used to manipulate and analyze data, and is a device that has the function of processing image data of plants and user emotion data.

[0584] "Imaging means" refers to a mechanism for visually capturing plant bodies, and is an image acquisition device that includes cameras and sensors.

[0585] "Identification means" refers to a device or program that has the function of analyzing captured data to identify the type of plant.

[0586] "Information generation means" refers to a device or program that has the function of creating relevant information based on the type of plant identified and providing it to the user.

[0587] A "risk assessment tool" is a device or program that uses generated information to analyze and evaluate the instability of the plant's condition and health.

[0588] "Emotional analysis means" refers to a device or program for analyzing emotional data collected from users and understanding its content.

[0589] A "recommendation tool" is a device or program that has the function of suggesting specific information or products to a user based on the user's emotional data and plant information.

[0590] "Communication means" refers to a device or program that has the function of transmitting evaluation results or additional information to an external control body or other system.

[0591] To implement this invention, it is necessary to construct a system that combines an information processing device, an emotion analysis engine, and a generative AI model. The user takes pictures of plants in a physical store using a terminal such as a smartphone or tablet. The terminal is equipped with a camera module and an audio input module, which are used to acquire image data of the plants and emotion data from the user's speech.

[0592] The terminal packages the collected data and sends it to the cloud server. To ensure security, the data is communicated using the HTTPS protocol. On the server, the image data is first analyzed using image recognition software (e.g., Google Cloud Vision API) to identify the type of plant. Based on this information, an information generation system creates information on how to cultivate the plant and provides recommendations.

[0593] Next, the server analyzes the user's emotional data using emotion analysis software (e.g., Microsoft Azure Emotion API). If the emotion is positive, the server uses a generative AI model to provide recommendations to the user. This information includes cultivation advice tailored to the plant's condition and appropriate related products.

[0594] For example, if a user takes a picture of a rose in the store and says "beautiful," the sentiment analysis system will analyze the positive emotion, and the recommendation system will suggest fertilizers and pruning tools suitable for growing roses. This entire sequence of information is also transmitted to an external control organization, contributing to store inventory management and improving customer satisfaction.

[0595] Example of a prompt:

[0596] "A user took a photo of a rose and commented, 'Beautiful.' Please generate a positive purchase suggestion regarding rose cultivation."

[0597] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0598] Step 1:

[0599] The user uses the device to photograph a plant and says "beautiful." Image and audio data are collected as input. The device packages this data and sends it to a cloud server. The HTTPS protocol is used for transmission, ensuring data security.

[0600] Step 2:

[0601] The server analyzes the plant using image recognition software (e.g., Google Cloud Vision API) based on the received image data. In this step, the image data is used as input for analysis, and information about the plant's species is obtained as output. Specific actions are then taken to identify the plant as a rose.

[0602] Step 3:

[0603] The server uses the received audio data to analyze the user's speech using emotion analysis software (e.g., Microsoft Azure Emotion API). In this step, the audio data is used as input for analysis, and the user's emotion data is obtained as output. Based on the analysis results, specific actions are taken to detect the user's positive emotions.

[0604] Step 4:

[0605] The server integrates plant species information and user sentiment data, and uses a generative AI model to generate recommendations for the user. Based on this input data, it generates advice and related product information as output. Specifically, it provides optimal methods for growing roses and recommends suitable products.

[0606] Step 5:

[0607] The server sends the generated recommendation information to the user's device. The transmitted information is displayed on the screen of the smartphone or tablet. Based on this information, the user can take specific actions such as considering how to care for plants or purchasing related products.

[0608] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0609] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0610] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0611] [Fourth Embodiment]

[0612] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0613] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0614] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0615] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0616] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0617] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0618] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0619] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0620] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0621] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0622] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0623] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0624] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0625] This invention is a system that aims to improve the management and safety of plants, such as street trees, by having users photograph them using an information processing device. Specifically, users take images of plants using a device such as a smartphone or tablet. In addition to the captured image data, the device collects location information and user identification information, and transmits it to a cloud server as a data package.

[0626] The server typically passes the received image data to an AI, which uses image recognition technology to identify the type of plant. This process utilizes a pre-trained model to analyze the image and determine whether it is a cherry tree or a pine tree. Based on the obtained plant information, the generating AI creates relevant information such as recommended care methods and appropriate distances from roads. This information is overlaid as text on the image.

[0627] Next, the server uses a risk assessment tool to evaluate the risk of fallen trees from the image data containing the generated information. The risk level is classified into ranks such as "high," "medium," and "low," and countermeasures are proposed as needed.

[0628] The server transmits these evaluation results to the management agency via communication channels. The management agency receives this information and plans appropriate measures for the plants deemed dangerous. The pruning plan created by the AI ​​is then implemented after final approval by the management agency.

[0629] For example, if a user takes a picture of a cherry tree in their neighborhood, the server recognizes the image data and determines that it is a cherry tree. The generated information will include phrases such as "sufficient distance from the road" and "maintenance recommended." Based on this result, the plant's risk level is assessed as "medium" and sent to the management agency. The management agency can then use this information to plan and carry out pruning within a few months.

[0630] In this way, by utilizing information provided by citizens, management agencies can efficiently and quickly manage the plants.

[0631] The following describes the processing flow.

[0632] Step 1:

[0633] Users take pictures of street trees using their smartphones or tablets. The device automatically acquires location information and records user identification information when the image is taken.

[0634] Step 2:

[0635] The device creates a data package for sending the captured image data to the cloud server. This package includes image data, location information, and user identification information.

[0636] Step 3:

[0637] The device uses HTTPS to securely send data packages to the cloud server. The user is notified when the communication is successful.

[0638] Step 4:

[0639] The server typically passes the received image data to the AI. The AI ​​usually uses a pre-trained model to analyze the types of plants in the image and determine specific types such as "cherry blossoms" or "pine trees."

[0640] Step 5:

[0641] Based on the identified plant species, the server's AI generates relevant information such as care instructions and the appropriate distance from roads. This generated information is overlaid as text on the image.

[0642] Step 6:

[0643] The server uses a risk assessment tool to evaluate the risk of fallen trees based on the generated information. The evaluation is divided into categories such as "high," "medium," and "low," and necessary countermeasures are generated.

[0644] Step 7:

[0645] The server transmits data summarizing the evaluation results and countermeasures to an external management organization via communication means. The transmitted information is then processed by the management organization.

[0646] Step 8:

[0647] The management agency reviews the received information and develops a concrete action plan based on proposals, including pruning plans created by the AI ​​generator. The plan is finalized after considering on-site surveys and other factors.

[0648] (Example 1)

[0649] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0650] In modern society, managing and ensuring the safety of plants is a crucial issue in preserving the urban environment. However, conventional methods are not sufficiently efficient for gathering information and assessing the risk of each individual plant, potentially leading to delays in management and safety risks. Therefore, a new system is needed to quickly and accurately grasp the condition of plants and formulate appropriate management plans.

[0651] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0652] In this invention, the server includes imaging and data collection means for users to photograph plants, acquiring location information and identification information using a device, and storing them as a data package; image recognition means using a trained model to analyze the data package and determine the type of plant; and information generation means using generative AI to generate relevant advice information based on the determined type of plant. As a result, information acquired by users can be quickly analyzed, and management organizations can obtain advice to achieve appropriate plant management and improve safety.

[0653] "User" refers to an individual or group that uses the system to photograph plants and provide data.

[0654] "Plant body" refers to street trees and other natural plants that are subject to management, including their species and the management of such plants.

[0655] "Image acquisition and data collection means" refers to the function that allows the user to photograph a plant and collect location information and identification information along with the image data using the device.

[0656] A "data package" refers to a dataset that combines captured image data, location information, and identification information into a single file.

[0657] "Image recognition methods using pre-trained models" refer to methods that analyze image data using already trained models to identify plant species.

[0658] "Information generation method using AI generation" refers to a process that generates advice information regarding management and care based on the type of plant identified using AI technology.

[0659] A "risk assessment method" refers to a technique for evaluating the safety of plants based on generated advisory information and determining the degree of risk, such as the risk of trees falling.

[0660] "Data transmission means" refers to methods for communicating obtained evaluation results and generated information to external parties such as management organizations.

[0661] A "management organization" refers to an organization responsible for the management, conservation, and safety improvement of plant bodies.

[0662] This invention is a system that uses an information processing device for the purpose of improving the management and safety of plants. The system begins with a user taking a photograph of a plant using a device such as a smartphone or tablet. The device acquires the captured image data, along with location information such as GPS and user identification information, and transmits this data package to a cloud server.

[0663] When the server analyzes the received data package, it typically uses a deep learning-based AI model to identify images. Specifically, it utilizes a pre-trained model such as a convolutional neural network (CNN) to determine the type of plant. It might then produce a result such as, "This is a cherry blossom."

[0664] Next, the server uses a generative AI model to generate relevant advice for the identified plant species. During this process, prompts are input, and the generative AI model, for example, responds to prompts such as "Tell me how to care for a cherry tree," by suggesting care methods and preventative measures. The generated information is overlaid as text on the image. Image editing is performed using software such as OpenCV or PIL.

[0665] Furthermore, the server uses a risk assessment tool to evaluate the risk of trees falling based on the generated information. This evaluation is quantified into ranks such as "high," "medium," and "low," which helps management organizations develop countermeasures as needed.

[0666] These evaluation results are transmitted to the management organization via the server. Based on this information, the management organization can develop regular maintenance and pruning plans, enabling quick and efficient plant management.

[0667] As a concrete example, if a user takes a photo of a cherry tree in a nearby park, the device sends that information to a cloud server. The server identifies it as a "cherry tree" and generates advice such as "the distance from the road is appropriate" and "maintenance is recommended." Based on this information, the level of risk is assessed as "medium," sent to the management organization, and pruning is scheduled within a few months. This helps maintain a safe and healthy urban environment.

[0668] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0669] Step 1:

[0670] The user takes a picture of a plant using their device. This involves the user taking a picture of the plant with a camera app and acquiring image data. The input is an image of the plant, and the output is an image file saved on the device.

[0671] Step 2:

[0672] The device collects location and user identification information based on the captured image. It uses the device's GPS function to obtain location information and extracts identification information from user settings. The input is location and identification information acquired at the time of image capture, and the output is a data package containing this information.

[0673] Step 3:

[0674] The terminal sends the generated data package to the cloud server. The data package is uploaded to the server via an internet connection. The input is the data package, and the output is the data received and stored on the server.

[0675] Step 4:

[0676] The server analyzes the received data package and identifies the type of plant. It analyzes images using a standard AI model based on deep learning. The input is image data within the data package, and the output is the identified plant type (e.g., "cherry blossom").

[0677] Step 5:

[0678] The server uses a generative AI model to generate information based on plant species. It takes a specific prompt as input to the generative AI and generates relevant care instructions and advice. The input is the plant species and the prompt, and the output is the generated information text.

[0679] Step 6:

[0680] The server overlays the generated information onto the image. An image editing library is used to place the obtained text information onto the image. The input is the generated information text and the original image data, and the output is a new image with the information displayed.

[0681] Step 7:

[0682] The server uses the generated information and risk assessment tools to evaluate the risk level of the plant. It utilizes an algorithm to calculate a risk rank (e.g., "High," "Medium," "Low"). The input is an image with generated information, and the output is the risk rank as the evaluation result.

[0683] Step 8:

[0684] The server transmits the evaluation results to the management organization. It uses communication methods to inform the management organization of the evaluation results and related information. The input is the risk level rank and related information, and the output is the management organization's confirmation of receipt.

[0685] (Application Example 1)

[0686] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0687] In urban areas and public spaces, falling trees and poorly managed plants pose safety threats. Unstable plants, in particular, can cause disasters under severe weather conditions. However, monitoring and managing plants requires significant resources, and efficient monitoring and appropriate countermeasures are needed. Furthermore, a system is needed where residents of the community actively share information and work together to maintain safety.

[0688] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0689] In this invention, the server includes imaging means for photographing plant bodies using an information processing device, identification means for identifying the type of plant body by analyzing the captured image data, and information adding means for generating information related to the plant body based on the identified type. This enables local residents to assess the risk level of plant bodies, quickly provide information on unstable plant bodies to management agencies, and efficiently carry out safety management.

[0690] An "information processing device" is an electronic device that has the function of photographing plant bodies and analyzing the image data.

[0691] "Imaging means" refers to a component that includes a camera function for photographing plant bodies.

[0692] "Identification means" refers to an algorithm or process for analyzing captured image data to identify the type of plant.

[0693] The "information addition means" is a function that generates and adds information related to a plant based on the type of plant identified.

[0694] A "risk assessment tool" is a function that uses additional information to evaluate the instability of a plant and determine its level of risk.

[0695] "Communication means" refers to a function for transmitting evaluation results and related information to an external management organization.

[0696] An "input method" is an interface for users to provide information about plants in the public sphere.

[0697] A "generation method" is a function that generates recommended actions to contribute to the safety management of the local community based on the information provided.

[0698] To implement this invention, a smartphone or tablet device is first used as an information processing device. The user holds this device and photographs plants in a public area. The captured image is acquired in high resolution by the camera built into the device and temporarily stored in the device's storage. Metadata such as the user's location information and identification information is also added to the image data.

[0699] After taking a picture, the device sends this data to a cloud server. The server uses image recognition technologies such as TensorFlow to analyze the received image data. This image recognition process identifies the type of plant, and based on the plant information identified by the "generative AI model," recommended care methods and information about the plant's instability are generated.

[0700] Furthermore, the server is equipped with a function to determine the level of risk. Using the generated information, the risk of trees falling is classified into numerical values ​​or ranks such as "high," "medium," and "low." These evaluation results are promptly transmitted to the management organization via cloud-based communication.

[0701] As a concrete example, suppose a citizen takes a picture of a cherry tree in a park with their smartphone and uploads the image to a cloud server via an application. The server analyzes the image data and recognizes it as a cherry tree. The AI ​​then generates a message such as "maintenance recommended," and the risk level is assessed as "medium," which is then reported to the management agency. Based on this information, the management agency plans pruning within a few months and takes action to maintain safety.

[0702] An example of a prompt message is as follows: "Location: 35.6895, 139.6917, Image: <Captured image data>, User ID: 123456, Please start analysis."

[0703] In this way, it can contribute to maintaining a safe and stable environment for the entire community.

[0704] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0705] Step 1:

[0706] The user takes an image of a plant using a device. The input consists of the plant image, location information, and user identification information. The device acquires this input, adds metadata (location information and user ID) to the image data, and temporarily stores it.

[0707] Step 2:

[0708] The terminal sends temporarily stored image data and metadata to the cloud server. The output is the transmission of a data package to the server. Communication takes place to confirm that the data has been successfully uploaded to the server.

[0709] Step 3:

[0710] The server receives the image data and performs preprocessing. In this step, the data is prepared in a format suitable for applying an image recognition algorithm (e.g., a TensorFlow model). This process results in output image data that can identify plants.

[0711] Step 4:

[0712] The server uses image recognition technology to identify the type of plant from the received image. The input is pre-processed image data, and the output is information about the identified plant type. Based on this, information corresponding to a specific plant is selected.

[0713] Step 5:

[0714] The server uses a generative AI model to generate information about care instructions and instability based on the identified plant species. The input is plant species information, and the output is the generated related information. This generated information is overlaid on the plant image.

[0715] Step 6:

[0716] The server uses the generated information to assess the risk level of the plants. The input is the generated relevant information, and the output is a risk level ranking (high, medium, low). This assessment is determined based on the condition of the identified plants.

[0717] Step 7:

[0718] The server transmits the risk assessment results and related information to the management agency. The input consists of the risk assessment results and plant identification information, while the output is the transmission of information to the management agency. After transmission, the management agency can obtain the basic information necessary to plan countermeasures.

[0719] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0720] This invention is a plant management system that combines an emotion engine with an information processing device, and aims to create richer data by analyzing the emotions a user experiences when photographing plants. Specific embodiments for carrying out this invention will be described below.

[0721] Users take photos of street trees or specific plants using devices such as smartphones or tablets. The system also collects user emotional data during photography through speech and facial recognition cameras.

[0722] The device generates a data package containing plant image data, location information, user identification information, and user sentiment data, and sends it to a cloud server. The HTTPS protocol is used to ensure data security.

[0723] The server typically uses AI to analyze the received image data and identify the type of plant. For example, suppose a cherry tree is identified from the image. Based on this information, the generating AI creates additional information such as appropriate care methods and the appropriate distance from the road, and overlays it as an image with this information.

[0724] Next, the emotion engine analyzes the user's emotional data, and if positive emotions are detected, it highlights recommended information about the plant. Conversely, if negative emotions are recognized, it provides more detailed safety information and care advice.

[0725] The server uses these results to assess the risk level of the plants and sends the assessment results along with the sentiment analysis data to the management organization. This enables the development of more empathetic and adaptive plant management plans.

[0726] For example, if a user takes a picture of a cherry tree and says "beautiful," the emotion engine will detect a positive emotion. Based on this, the server will generate information highlighting the recommended time to view the cherry blossoms and simple maintenance tips. This information will be sent to the relevant authorities and can contribute to providing safety information to local residents and promoting tourism.

[0727] Thus, the system of the present invention enables advanced data provision and management by combining machine learning and emotion analysis in plant management.

[0728] The following describes the processing flow.

[0729] Step 1:

[0730] Users take photos of street trees or specific plants using their smartphones or tablets. During this process, the system captures images using the device's camera and also collects user emotion data using voice input and facial recognition features.

[0731] Step 2:

[0732] The device combines captured image data, location information, user identification information, and sentiment data into a data package. This data package is then prepared for transmission to a cloud server.

[0733] Step 3:

[0734] The device uses the HTTPS protocol to send data packages to the cloud server via secure communication. Once successful communication is confirmed, the user is notified.

[0735] Step 4:

[0736] The server typically passes the received image data to the AI, which then begins image analysis. This analysis uses a pre-trained model to identify the type of plant. For example, it might be identified as a cherry blossom.

[0737] Step 5:

[0738] Based on the identified plant species, the server uses a generative AI to generate relevant information such as maintenance methods and appropriate distances from roads, and integrates this information into the image as an overlay.

[0739] Step 6:

[0740] The server uses an emotion engine to analyze the emotional data provided by the user. When positive emotions are detected, a function is activated to highlight recommended information about the plant. On the other hand, if negative emotions are recognized, more detailed safety information is provided.

[0741] Step 7:

[0742] The server evaluates the risk level of the plant based on the generated information and the results of sentiment analysis. The risk level is clearly indicated by a ranking system of "high," "medium," and "low."

[0743] Step 8:

[0744] The server transmits the evaluation results and related information to an external management organization via communication means. The management organization then develops a plant management plan based on the received information.

[0745] Through this series of processes, we will build a system that provides information and plans to ensure the stability and safety of plants instantly, based on simple input from the user.

[0746] (Example 2)

[0747] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0748] Conventional plant management systems often identify plant species and provide basic care information, but they do not support personalized information delivery that takes user emotions into account. As a result, the emotional state in which users observe plants is not adequately reflected, sometimes leading to inappropriate plant management plans. Furthermore, the generation of information about plants is limited, resulting in a lack of data that can be used for plant management across an entire region. Therefore, there is a need for a system that utilizes user emotional data to enrich their experience while contributing to regional plant management.

[0749] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0750] In this invention, the server includes data acquisition means for photographing plants and collecting emotional data from the user's voice and facial expressions; identification means for analyzing the captured image data and identifying the type of plant; information generation means for generating information related to the plant using a generative artificial intelligence model based on the identified type and the user's emotional data; emotion analysis means for analyzing the emotional data, highlighting specific information when positive emotions are detected, and providing detailed safety information when negative emotions are detected; risk determination means for evaluating the instability of the plant using the generated information; and communication means for transmitting the evaluation results and emotional analysis results to an external management organization. This enables the provision of appropriate information according to the user's emotional state, and allows for more effective data analysis and planning for regional plant management.

[0751] The "data acquisition means" is a mechanism that collects emotional data through image data, the user's voice, and facial expressions when the user photographs a plant.

[0752] "Identification means" refers to a device that analyzes image data acquired by data acquisition means and has the function of identifying the type of plant.

[0753] The "information generation means" is a device for creating additional information related to a plant using a generative artificial intelligence model, based on the identified plant type and the user's emotional data.

[0754] An "emotion analysis tool" is a mechanism that analyzes emotional data obtained from users and adjusts the way information is displayed according to whether the emotions are positive or negative.

[0755] A "risk assessment method" is a means of evaluating the instability of a plant based on the generated information and determining its level of risk.

[0756] "Communication means" refers to the means of transmitting evaluation results and sentiment analysis data to an external management organization.

[0757] The system of this invention is designed to allow users to effectively manage plants using mobile devices such as smartphones and tablets. Users collect image data by taking pictures of street trees or specific plants. The device is equipped with a speech recognition function and a facial recognition camera, and by analyzing the user's voice and facial data, it acquires the user's emotional data.

[0758] The device integrates acquired image data, location information, user identification information, and sentiment data, and sends it to a cloud server as a data package. Data security is ensured by using the HTTPS protocol.

[0759] The server uses AI technology to analyze the received image data and identify the type of plant. This analysis utilizes a generative AI model to generate information on care methods and location based on the identified plant. Furthermore, an emotion engine analyzes the user's emotional data, highlighting information when positive emotions are recognized. Conversely, in the case of negative emotions, it provides detailed safety information and care advice.

[0760] For example, when a user takes a picture of a cherry tree and says "beautiful," the emotion engine detects the positive emotion, and the server generates information highlighting the best time to view the cherry blossoms and maintenance tips. This information is sent to the management organization and contributes to providing safety information to local residents and promoting tourism. An example of a prompt to the generating AI model would be, "Generate maintenance instructions from the photo of the cherry tree, and if the user's emotion is positive, highlight the recommended information."

[0761] This system allows users to obtain high-quality information tailored to their emotions, potentially adding new value to local plant management.

[0762] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0763] Step 1:

[0764] The user takes a picture of the plant using a device such as a smartphone or tablet. The input is an image of the plant. The device uses a speech and facial recognition camera to collect emotional data from the user's voice and facial expressions. The output of this step is image data and emotional data. Specifically, the user uses the device's camera to take a picture of the plant, and at the same time, emotional input is provided through voice and facial expressions.

[0765] Step 2:

[0766] The device integrates captured image data, location information, user identification information, and sentiment data to generate a data package. The various data acquired in Step 1 are used as input. Data processing involves formatting this information into a single data package. The output is a data package transmitted via HTTPS. Specifically, the device organizes the data and prepares it for transmission to the cloud server via a communication module.

[0767] Step 3:

[0768] The server receives a data package sent from the terminal. The input includes the received image data. The server uses AI technology to analyze the image data and identify the type of plant. The data processing involves extracting features from the plant image and comparing them with an existing database to identify the type. The output is information about the identified plant type. Specifically, the AI ​​engine processes the image and obtains the identification result.

[0769] Step 4:

[0770] The server uses a generative AI model to generate information related to the plant based on the identified plant type and the user's emotional data. The inputs used are the plant type information and emotional data obtained in step 3. As data processing, this information is used to generate care instructions and recommendations. The output is the generated additional information. Specifically, the AI ​​model generates data such as care know-how and viewing times.

[0771] Step 5:

[0772] The server analyzes emotional data, highlighting specific information when positive emotions are detected and providing detailed safety information when negative emotions are detected. The input is the user's emotional data, which is then analyzed. Data processing involves customizing the information based on the intensity of the emotions. The output is customized plant information. Specifically, the system adjusts how the generated information is displayed.

[0773] Step 6:

[0774] Ultimately, the server transmits all analysis results and sentiment analysis data to the management agency for use in plant management across the region. The information and analysis results obtained in step 5 are used as input. A communication module is used to transmit data to the management agency. The output is the agency's receipt and use of the information. Specifically, the information is integrated into the administrator's interface to aid in the development of regional plans.

[0775] (Application Example 2)

[0776] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0777] Modern plant management systems require accurate identification of plant species and conditions, but a problem is that they fail to consider the crucial information of user emotions. Furthermore, there is a lack of methods to provide individually customized information and suggestions about plants to enrich the customer experience in retail stores. Therefore, there is a need for information delivery methods that respond to user emotions.

[0778] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0779] In this invention, the server includes, via an information processing device, an imaging means for capturing a plant; an identification means for identifying the type of plant by analyzing the captured image data; an information generation means for generating information related to the plant based on the identified type; a risk assessment means for evaluating the instability of the plant using the generated information; an emotion analysis means for collecting and analyzing emotion data from the user; a recommendation means for providing specific suggestions for additional information or items related to the plant based on the user's emotion data; and a communication means for transmitting the evaluation results and additional information to an external control organization. This enables comprehensive management of plants and the provision of customized information.

[0780] An "information processing device" is an electronic device used to manipulate and analyze data, and is a device that has the function of processing image data of plants and user emotion data.

[0781] "Imaging means" refers to a mechanism for visually capturing plant bodies, and is an image acquisition device that includes cameras and sensors.

[0782] "Identification means" refers to a device or program that has the function of analyzing captured data to identify the type of plant.

[0783] "Information generation means" refers to a device or program that has the function of creating relevant information based on the type of plant identified and providing it to the user.

[0784] A "risk assessment tool" is a device or program that uses generated information to analyze and evaluate the instability of the plant's condition and health.

[0785] "Emotional analysis means" refers to a device or program for analyzing emotional data collected from users and understanding its content.

[0786] A "recommendation tool" is a device or program that has the function of suggesting specific information or products to a user based on the user's emotional data and plant information.

[0787] "Communication means" refers to a device or program that has the function of transmitting evaluation results or additional information to an external control body or other system.

[0788] To implement this invention, it is necessary to construct a system that combines an information processing device, an emotion analysis engine, and a generative AI model. The user takes pictures of plants in a physical store using a terminal such as a smartphone or tablet. The terminal is equipped with a camera module and an audio input module, which are used to acquire image data of the plants and emotion data from the user's speech.

[0789] The terminal packages the collected data and sends it to the cloud server. To ensure security, the data is communicated using the HTTPS protocol. On the server, the image data is first analyzed using image recognition software (e.g., Google Cloud Vision API) to identify the type of plant. Based on this information, an information generation system creates information on how to cultivate the plant and provides recommendations.

[0790] Next, the server analyzes the user's emotional data using emotion analysis software (e.g., Microsoft Azure Emotion API). If the emotion is positive, the server uses a generative AI model to provide recommendations to the user. This information includes cultivation advice tailored to the plant's condition and appropriate related products.

[0791] For example, if a user takes a picture of a rose in the store and says "beautiful," the sentiment analysis system will analyze the positive emotion, and the recommendation system will suggest fertilizers and pruning tools suitable for growing roses. This entire sequence of information is also transmitted to an external control organization, contributing to store inventory management and improving customer satisfaction.

[0792] Example of a prompt:

[0793] "A user took a photo of a rose and commented, 'Beautiful.' Please generate a positive purchase suggestion regarding rose cultivation."

[0794] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0795] Step 1:

[0796] The user uses the device to photograph a plant and says "beautiful." Image and audio data are collected as input. The device packages this data and sends it to a cloud server. The HTTPS protocol is used for transmission, ensuring data security.

[0797] Step 2:

[0798] The server analyzes the plant using image recognition software (e.g., Google Cloud Vision API) based on the received image data. In this step, the image data is used as input for analysis, and information about the plant's species is obtained as output. Specific actions are then taken to identify the plant as a rose.

[0799] Step 3:

[0800] The server uses the received audio data to analyze the user's speech using emotion analysis software (e.g., Microsoft Azure Emotion API). In this step, the audio data is used as input for analysis, and the user's emotion data is obtained as output. Based on the analysis results, specific actions are taken to detect the user's positive emotions.

[0801] Step 4:

[0802] The server integrates plant species information and user sentiment data, and uses a generative AI model to generate recommendations for the user. Based on this input data, it generates advice and related product information as output. Specifically, it provides optimal methods for growing roses and recommends suitable products.

[0803] Step 5:

[0804] The server sends the generated recommendation information to the user's device. The transmitted information is displayed on the screen of the smartphone or tablet. Based on this information, the user can take specific actions such as considering how to care for plants or purchasing related products.

[0805] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0806] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0807] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0808] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0809] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0810] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0811] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0812] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0813] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0814] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0815] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0816] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0817] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0818] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0819] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0820] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0821] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0822] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0823] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0824] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0825] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0826] The following is further disclosed regarding the embodiments described above.

[0827] (Claim 1)

[0828] An information processing device provides an imaging means for photographing plant bodies,

[0829] An identification means for identifying the type of plant by analyzing the captured image data,

[0830] Information addition means for generating information related to the plant body based on the identified type,

[0831] A risk assessment means for evaluating the instability of the plant using the aforementioned added information,

[0832] A communication means for transmitting the aforementioned evaluation results to an external management organization,

[0833] A system that includes this.

[0834] (Claim 2)

[0835] The system according to claim 1, wherein the information generation means generates information including the physical distance to the plant and suggestions for care methods.

[0836] (Claim 3)

[0837] The system according to claim 1, wherein the communication means transmits a proposed work plan based on the evaluation results to an external management organization.

[0838] "Example 1"

[0839] (Claim 1)

[0840] An imaging and data collection means that allows the user to photograph a plant, acquire location information and identification information using a device, and store it as a data package,

[0841] The aforementioned data package is analyzed and an image recognition means using a trained model for identifying the type of plant is provided.

[0842] Information generation means utilizing a generation AI that generates relevant advisory information based on the type of plant identified,

[0843] A risk assessment means for integrating the generated information into an image and evaluating the stability and safety of the plant body,

[0844] A data transmission means for communicating the aforementioned evaluation results to an external management organization,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] The system according to claim 1, wherein the information generation means generates information including advice on appropriate distances and maintenance methods for plant bodies.

[0848] (Claim 3)

[0849] The system according to claim 1, wherein the data transmission means proposes a countermeasure plan based on the evaluation results to an external management organization.

[0850] "Application Example 1"

[0851] (Claim 1)

[0852] An information processing device provides an imaging means for photographing plant bodies,

[0853] An identification means for identifying the type of plant by analyzing the captured image data,

[0854] Information addition means for generating information related to the plant body based on the identified type,

[0855] A risk assessment means for evaluating the instability of the plant using the aforementioned added information,

[0856] A communication means for transmitting the aforementioned evaluation results to an external management organization,

[0857] An input method that allows users to easily provide information about plants in the public sphere,

[0858] A generation means for generating recommended actions to contribute to the safety management of the local community based on the information provided above,

[0859] A system that includes this.

[0860] (Claim 2)

[0861] The system according to claim 1, wherein the information-adding means generates information including the physical distance to the plant and suggestions for care methods, and makes it public as part of local safety management.

[0862] (Claim 3)

[0863] The system according to claim 1, wherein the communication means includes a function of transmitting a proposed work plan based on the evaluation results to an external management organization and notifying local residents of the same.

[0864] "Example 2 of combining an emotion engine"

[0865] (Claim 1)

[0866] A data acquisition method that photographs plants and collects emotional data from the user's voice and facial expressions,

[0867] An identification means that analyzes captured image data to identify the type of plant,

[0868] Information generation means that generates information related to plants using a generative artificial intelligence model based on identified species and user sentiment data,

[0869] An emotion analysis means that analyzes emotion data, highlights specific information when positive emotions are detected, and provides detailed safety information when negative emotions are detected,

[0870] A risk assessment method that evaluates the instability of a plant using the generated information,

[0871] A means of communication for transmitting evaluation results and sentiment analysis results to an external management organization,

[0872] A system that includes this.

[0873] (Claim 2)

[0874] The system according to claim 1, which generates information including the physical distance to a plant and suggestions for care methods, and adjusts the method of displaying the information based on emotional data.

[0875] (Claim 3)

[0876] The system according to claim 1, which transmits a proposed work plan based on evaluation results and sentiment analysis data to an external management organization.

[0877] "Application example 2 when combining with an emotional engine"

[0878] (Claim 1)

[0879] The information processing device includes an imaging means for capturing plant bodies,

[0880] An identification means for identifying the type of plant by analyzing the captured image data,

[0881] Information generation means for generating information related to the plant body based on the identified type,

[0882] Using the generated information, a risk assessment means for evaluating the instability of the plant body,

[0883] A sentiment analysis method that collects and analyzes sentiment data from users,

[0884] A recommendation system that provides additional information about plants or specific suggestions for items based on user sentiment data,

[0885] A communication means for transmitting the aforementioned evaluation results and additional information to an external control body,

[0886] A system that includes this.

[0887] (Claim 2)

[0888] The system according to claim 1, wherein the information generation means generates information including physical spacing to plants and suggestions for maintenance methods.

[0889] (Claim 3)

[0890] The system according to claim 1, wherein the communication means transmits a proposed work plan based on the evaluation results to an external control organization. [Explanation of Symbols]

[0891] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An information processing device provides an imaging means for photographing plant bodies, An identification means for identifying the type of plant by analyzing the captured image data, Information addition means for generating information related to the plant body based on the identified type, A risk assessment means for evaluating the instability of the plant using the aforementioned added information, A communication means for transmitting the aforementioned evaluation results to an external management organization, A system that includes this.

2. The system according to claim 1, wherein the information generation means generates information including the physical distance to the plant and suggestions for maintenance methods.

3. The system according to claim 1, wherein the communication means transmits a proposed work plan based on the evaluation results to an external management organization.

Images