system

Abstract

We provide the system. [Solution] Means for preprocessing the visual data of the monitored object received from the acquisition device, Means for identifying an object based on pre-processed visual data and determining whether the identified object is located in a predetermined prohibited area, A means of notifying relevant organizations of information about the object identified based on the judgment result, A means of notifying the user in real time about objects within a designated restricted area via a digital terminal and guiding them to an appropriate parking location. A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Illegal parking in urban areas poses a major obstacle to pedestrians, especially those with physical disabilities, and is a factor that impairs the safety and comfort of the city. In response to this problem, efficient and real-time monitoring and prompt response are required, but it is difficult to respond sufficiently with the current manual monitoring and reporting systems. Therefore, it is necessary to introduce a system that improves monitoring efficiency and quickly detects and reports illegal parking.

Means for Solving the Problems

[0005] This invention provides a system that preprocesses video data received from an acquisition device and detects bicycles using object recognition technology. This system can automatically identify bicycles in designated prohibited areas and immediately notify relevant authorities based on the results. As a result, it becomes possible to effectively and quickly monitor and report illegal parking, thereby improving urban safety and comfort.

[0006] "Acquisition device" refers to equipment such as cameras and sensors that capture video data and transmit it to a server.

[0007] "Video data" refers to visual information transmitted from an acquisition device, which is used for object identification and location determination.

[0008] "Preprocessing" refers to the process of optimizing the quality and format of video data to make it suitable for analysis.

[0009] "Object recognition" refers to the technology of detecting and recognizing specific objects within video data.

[0010] A "prohibited area" refers to a specific area defined by a geographic information database where parking bicycles is prohibited by law or regulations.

[0011] "Determination" refers to the process of evaluating whether the location of the identified object is within the prohibited area.

[0012] "Notification" refers to the act of transmitting the judgment results to the relevant organizations, and is carried out using email or APIs.

[0013] "Related organizations" refers to public institutions such as the police and city hall that are responsible for taking measures against illegal parking. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

MODE FOR CARRYING OUT THE INVENTION

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0018] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention is a system that analyzes video data from an acquisition device, with the aim of rapidly and effectively detecting and reporting illegal parking. The system utilizes AI-powered object recognition technology and a database-based method for determining prohibited areas.

[0036] The server first receives video data from the acquisition device and performs preprocessing on that data. This preprocessing includes noise reduction and image resizing. Next, the server passes the preprocessed data to an AI algorithm for object recognition. This determines where objects such as bicycles are located within the video.

[0037] Subsequently, the server checks the location information of the identified bicycle against a database of prohibited areas to determine if it is illegally parked. Once this determination is made, the server records the information of the illegally parked bicycle and initiates the process of notifying the relevant authorities.

[0038] As a concrete example, imagine a camera installed in a busy area of ​​a city that continuously captures video in real time. The server uses AI to detect bicycles parked at specific locations from this video, and if they are within a designated prohibited area, it automatically records that fact. The server then notifies the police or city hall via email to prompt action against illegal parking.

[0039] The terminal provides an interface that allows users to check information on illegally parked bicycles in real time, helping the general public to park their bicycles responsibly and with awareness.

[0040] Thus, this system is effective in addressing the problem of illegal bicycle parking in urban areas and contributes to improving public safety and comfort.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The server receives video data in real time from the acquisition device. This video data is transmitted from cameras designed to continuously cover the area being monitored.

[0044] Step 2:

[0045] The server performs preprocessing on the received video data. Specifically, it removes noise from the video and resizes the resolution to a size suitable for the AI ​​algorithm.

[0046] Step 3:

[0047] The server inputs the pre-processed video into an AI algorithm and begins object recognition. Here, a convolutional neural network (CNN) is used to detect bicycles in the image and determine their location.

[0048] Step 4:

[0049] The server then compares the location information of the identified bicycle with a geographic information database. This allows it to determine whether the identified bicycle is located within a no-parking zone.

[0050] Step 5:

[0051] The server records information in its database if it determines that a bicycle is illegally parked. This record includes the date and time of detection, location information, and a video snapshot.

[0052] Step 6:

[0053] The server automatically generates a notification message based on recorded illegal parking information and sends the notification to the relevant authorities in the appropriate format. This notification is sent via email or a dedicated API.

[0054] Step 7:

[0055] The terminal provides a user interface that allows users to view information on illegally parked bicycles. This enables users to understand the situation of illegal parking in real time and to be more mindful of preventing parking problems.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] Illegal parking of bicycles poses a significant problem for urban traffic, landscape, and crime prevention. Conventional measures require substantial human resources for monitoring and enforcement, making effective control difficult. This invention aims to provide a system that efficiently and quickly detects illegally parked bicycles with limited resources and notifies the relevant authorities.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for performing object identification using a generated AI model with the preprocessed video data, determining the location of the identified object, and determining whether the location matches a predetermined prohibited area database, and means for notifying relevant organizations of information about the identified object based on the determination result via electronic communication means. This makes it possible to automatically detect illegal parking and notify relevant parties quickly and with high accuracy.

[0061] An "acquisition device" is a device used to acquire video data from an object being observed, and includes devices such as cameras and sensors.

[0062] "Preprocessing" refers to the initial processing of received video data, such as noise reduction and resizing.

[0063] A "generative AI model" is a model that uses machine learning algorithms and is a mathematical representation for performing data analysis or object recognition in specific tasks.

[0064] "Object recognition" refers to the technology of detecting and classifying specific objects contained within image data.

[0065] A "restricted area database" is a database that holds information about restrictions in a specific area, including information about geographical boundaries.

[0066] "Electronic communication methods" refer to technologies and protocols used for digital communication via email or networks.

[0067] "Geographic information management technology" refers to technologies for collecting, managing, and analyzing geographic information, and includes GIS (Geographic Information Systems).

[0068] In implementing this invention, the server is primarily responsible for data reception, processing, and notification. The server receives video data in real time from cameras installed throughout the city as acquisition devices. These cameras have appropriate image quality and frame rates, enabling surveillance at various points in the city.

[0069] The server preprocesses the received video data using image processing libraries such as OpenCV. This preprocessing includes noise reduction, image quality improvement, and resolution modification. This processing improves the accuracy of object recognition by the subsequent generative AI model.

[0070] Next, the server applies machine learning algorithms such as YOLO (You Only Look Once) as a "generative AI model." This model quickly and accurately identifies objects such as bicycles present in the video. Then, by extracting the location data of the identified objects, it identifies where illegal parking is occurring.

[0071] The server uses geographic information management technologies such as GIS to compare identified location data with a database of prohibited areas. This database includes prohibited areas pre-defined based on urban planning and transportation policies. This allows the server to immediately determine whether or not illegal parking is present.

[0072] If the assessment confirms illegal parking, the server will use electronic communication methods to send a notification to the relevant authorities via email or a dedicated application. This notification will include the location information of the illegally parked bicycle and related image data, enabling a swift response.

[0073] Furthermore, the terminal provides an interface that allows users to check information on illegally parked bicycles in real time. This terminal enables users to understand the current state of illegal parking and contribute to improving parking etiquette.

[0074] A concrete example would be a process where cameras installed in the city center continuously transmit video footage to a server, which then analyzes the data to immediately identify illegal parking at specific locations and notifies the police and public transport authorities.

[0075] An example of a prompt is, "Explain the procedure for detecting objects from real-time video and comparing them to prohibited areas," which demonstrates an application using a generative AI model.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The server receives video data from the acquisition device. The input is real-time video data from the camera, and by acquiring this data, the server is ready to begin detecting illegally parked bicycles. Specifically, it continuously receives digital video signals via the network.

[0079] Step 2:

[0080] The server performs preprocessing on the received video data. The input is the received video data, and the output is image data with noise removed and the resolution appropriately adjusted. Here, the OpenCV library is used to perform noise reduction and image resizing, thereby preparing the data for subsequent analysis.

[0081] Step 3:

[0082] The server inputs preprocessed image data into a generating AI model to perform object recognition. The input is preprocessed image data, and the output is data identifying the position of bicycles within the image. Specifically, the YOLO model is applied to detect bicycles in each frame and extract their coordinate information.

[0083] Step 4:

[0084] The server uses geographic information management technology (GIS) to compare the location information of identified objects against a prohibited area database. The input is the identified location information, and the output is the result of determining whether the object is within a prohibited area. In this process, the location information is cross-referenced using GIS to determine the possibility of illegal parking.

[0085] Step 5:

[0086] Based on the judgment results, the server notifies the relevant authorities of information about bicycles deemed illegally parked. The input is the result of matching against prohibited areas, and the output is detailed information about the reported illegally parked bicycles. Specifically, it automatically notifies the necessary authorities via email or API and transmits illegal parking information, including image data.

[0087] Step 6:

[0088] The terminal provides users with real-time information on illegally parked bicycles. Input is the reported information received from the server, and output is the illegal parking status displayed on the user interface. This process presents information in a visually easy-to-understand format, allowing users to check it immediately.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] Illegal bicycle parking in urban areas is a major problem for city life, as it obstructs traffic, damages the aesthetics of the area, and hinders emergency evacuation routes. However, the current monitoring system suffers from a shortage of personnel and limitations in monitoring area, making it difficult to quickly detect and respond to illegal parking. Furthermore, there is a lack of guidance for citizens to choose appropriate parking locations. Therefore, there is a need to establish a system that enables real-time detection and reporting of illegal parking, as well as guidance to citizens on suitable parking locations.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for preprocessing visual data of a monitored object received from an acquisition device; means for identifying an object based on the preprocessed visual data and determining whether the identified object is located in a predetermined prohibited area; means for notifying relevant organizations of information about the identified object based on the determination result; and means for notifying users of a digital terminal in real time about objects located within the predetermined prohibited area and guiding them to an appropriate parking location. This enables rapid detection and response to illegal parking and guidance of citizens to appropriate parking locations.

[0094] An "acquisition device" is hardware or a system used to acquire visual data of a monitored object.

[0095] "Preprocessing" refers to the process of removing noise and adjusting the size of acquired visual data to convert it into a state suitable for analysis.

[0096] "Object recognition" is a technology that uses algorithms such as AI to detect specific objects present in visual data and to identify what they are.

[0097] A "restricted area" is information indicating a specific area where the presence of objects is not permitted, and this information is cross-referenced with a geographic information database.

[0098] "Related organizations" refer to organizations or entities that receive information about illegal parking and take necessary action.

[0099] "Notification" is the act of transmitting information about an identified object to a designated recipient using electronic means of communication.

[0100] A "digital device" is an electronic device that provides a user interface, and includes smartphones and tablets.

[0101] "Real-time" refers to the ability to process or provide data instantly, without delay.

[0102] "Bicycle parking location guidance" is a service that provides users with information and guidance on appropriate bicycle parking locations.

[0103] This invention comprises a system for detecting and notifying illegal bicycle parking using a digital terminal. The server receives visual data from acquisition devices installed in the monitored area and performs preprocessing. This preprocessing includes noise reduction and image resizing. As a result, the data is appropriately analyzed by an AI algorithm, enabling object identification.

[0104] The server uses TENSORFLOW® (registered trademark) or similar technologies to perform object identification and detect objects such as bicycles included in the visual data. The identified objects are then compared with prohibited area information stored in the geographic information database, and if a match is found, it is determined to be illegally parked. Information regarding illegal parking is immediately notified to the relevant authorities via electronic communication.

[0105] At the same time, users with digital devices can receive notifications in real time. The devices display on-site information about illegally parked bicycles and also provide guidance to appropriate parking locations. For example, if a citizen tries to park their bicycle near the entrance of a park, the device can issue an alert saying, "This is a no-parking zone," and display information about the nearest parking area along with a map.

[0106] The aim is to enable rapid response to illegal bicycle parking in urban areas and appropriate parking guidance through the user experience. In the system construction based on this invention, object recognition using a generative AI model and instant communication functions play important roles.

[0107] Examples of input prompts for a generative AI model for specific scenarios are as follows:

[0108] "Please explain how to identify bicycles parked in restricted areas using camera footage and generate an alert message when they are detected."

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The server receives visual input data from the acquisition device. The received data is acquired as a raw video stream. This data is used as input for processing in the next step.

[0112] Step 2:

[0113] The server performs preprocessing on the received visual data. This preprocessing includes removing noise from the image and resizing it to a size that is easily handled by the AI ​​model. As a result of the preprocessing, appropriately formatted video data is output.

[0114] Step 3:

[0115] The server analyzes the pre-processed video data using an AI model to perform object identification. Specifically, the AI ​​model detects specific objects in the video, such as bicycles. The obtained identification information is generated as output and used in subsequent steps.

[0116] Step 4:

[0117] The server compares the location information of the identified object with a geographic information database. This comparison determines whether the object is located within a no-parking zone. The result of the determination (whether or not illegal parking is present) and the object's location information are output.

[0118] Step 5:

[0119] If illegal parking is detected, the server notifies the relevant authorities of the determination. The notification is made automatically using electronic communication methods, and the content of the report is output.

[0120] Step 6:

[0121] The device notifies the user in real time about objects within the restricted area. The notification is sent to the device, and guidance to a suitable parking location is also provided. This notification allows the user to immediately understand the current situation in the restricted area and take appropriate action.

[0122] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0123] This invention provides a more interactive and comprehensive management system by combining an illegal parking detection and notification system with an emotion engine that recognizes user emotions. This system operates by integrating object recognition capabilities using video data with emotion analysis.

[0124] The server preprocesses the video data received from the acquisition device and uses an AI algorithm to detect bicycles. The server then compares the location of the detected bicycles with a geographic information database to determine if they are illegally parked in a prohibited area. If illegal parking is confirmed, the server notifies the relevant authorities based on the recorded information.

[0125] This system also features an emotion engine that identifies emotions from the user's facial expressions and voice. When a user interacts with the system, the terminal analyzes the user's emotions in real time and records them in a database. For example, if a user looking for a place to park their bicycle is feeling stressed, the emotion engine can identify that emotion and provide information about available parking locations.

[0126] As a concrete example, consider a scenario where a user is using a device to search for a bicycle parking space. In this case, the system analyzes the user's voice and facial expressions using an emotion engine. If it detects anxiety or impatience, the device displays a map and guides the user to nearby available parking spaces. This information can also be shared with relevant organizations for safety management purposes.

[0127] In this way, this system, which incorporates an emotion engine, achieves both a better user experience and improved public safety by integrating object detection and emotion analysis.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The server receives video data in real time from the acquisition device. This data captures the situation within the monitoring area.

[0131] Step 2:

[0132] The server performs preprocessing on the received video data. Specifically, it removes noise from the video and adjusts the image to a size that is easy to analyze.

[0133] Step 3:

[0134] The server inputs the pre-processed video data into an AI algorithm to perform object recognition. Here, it detects bicycles and other objects and determines their location.

[0135] Step 4:

[0136] The server then compares the location information of the identified object with a geographic information database to determine whether it is within a restricted area.

[0137] Step 5:

[0138] The server notifies the relevant authorities of details about objects determined to be illegally parked. This notification is made promptly via email or API.

[0139] Step 6:

[0140] The device captures the user's facial expressions and voice and sends them to the emotion engine. This is to analyze the user's emotions in real time while they are interacting with the system.

[0141] Step 7:

[0142] The server analyzes emotional data obtained from the emotion engine and generates feedback tailored to the user's state. For example, if the user is feeling stressed, it will present supportive information.

[0143] Step 8:

[0144] The device displays necessary information to the user based on emotional data. This includes providing specific guidance on areas where bicycles can be parked, as well as messages aimed at addressing the user's emotional needs.

[0145] Through this process, the system can not only manage illegal parking but also respond to users' emotions, aiming to maintain a more interactive and effective urban environment.

[0146] (Example 2)

[0147] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0148] In modern urban environments, illegal bicycle parking is a significant problem that severely impairs public traffic and aesthetics. However, simply monitoring parking is not enough; responses that consider the feelings of users are also required. Therefore, in addition to detecting illegal parking, it is necessary to understand the feelings of users and provide appropriate information accordingly. A comprehensive system is needed to address these complex issues.

[0149] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0150] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for identifying an object based on the preprocessed video data and determining whether the identified object is located in a predetermined prohibited area, and means for analyzing the user's facial expressions and voice to identify their emotions. This enables the detection of illegal parking and the provision of information based on the user's emotions.

[0151] An "acquisition device" is a device that acquires video data from the area being monitored.

[0152] "Video data" refers to data that digitally records the visual information of the subject being monitored.

[0153] "Preprocessing" refers to the initial stage of data processing performed to make video data easier to analyze.

[0154] "Object" refers to an identifiable entity within video data, and in this invention, it primarily refers to a bicycle.

[0155] "Identification" refers to the act of identifying the type and characteristics of an object based on video data.

[0156] A "forbidden area" refers to a designated location where the presence of objects is not permitted.

[0157] "Determination" is the act of determining whether or not an identified object exists in a prohibited area.

[0158] "Relevant body" refers to any organization or body that has the authority to take appropriate action upon receiving notification.

[0159] "Notification" refers to the act of transmitting information, such as judgment results, to other organizations or devices.

[0160] "Facial expressions and voice" refer to the user's visual and auditory expressions of emotion.

[0161] "Emotions" are elements that indicate the user's psychological state and are identified by the system.

[0162] "Information provision" refers to the act of presenting useful data to users.

[0163] This invention integrates a module that analyzes user emotions into a system for detecting and reporting illegally parked bicycles. The system incorporates both object detection and emotion recognition capabilities, aiming to improve the user experience and maintain public order.

[0164] The server receives video data from acquisition devices such as surveillance cameras. This data is preprocessed using the OpenCV library, including noise reduction and resolution adjustment. The preprocessed data is then analyzed by AI algorithms utilizing TensorFlow, PyTorch, etc., to detect bicycles in the video. The location information of the detected objects is cross-referenced with a geographic information database to evaluate whether they fall within a prohibited area. Database tools such as PostGIS are used for this process. If parking in a prohibited area is confirmed, the server automatically notifies the relevant authorities. The notification is securely transmitted via email or application programming interface.

[0165] On the other hand, when a user interacts with the device, the device uses its camera and microphone to capture the user's facial expressions and voice in real time. By analyzing this data using sentiment analysis APIs such as Google® Cloud Vision and IBM Watson®, the system identifies the user's emotions. For example, if the system detects impatience or anxiety while a user is looking for a bicycle parking space, the device uses the Google Maps API to guide the user to the nearest available parking space.

[0166] As a concrete example, consider a situation where a user is looking for a bicycle parking space near a train station. The system detects the user's voice and facial expressions, and if it detects "anxiety," the terminal displays "We will guide you to a nearby bicycle parking space" on the screen and provides a function to suggest the nearest possible locations. In this case, the system will respond if the user says, "Tell me about nearby bicycle parking spaces."

[0167] This configuration allows the system to check for parking violations in real time while providing services tailored to the user's emotions, creating a more efficient and user-friendly environment.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The server receives video data in real time from the acquisition device. This data is preprocessed using the OpenCV library. The input is video data from the surveillance camera, and the output is high-quality video data with noise reduction. Specifically, filtering is performed to reduce shadows and noise.

[0171] Step 2:

[0172] The server inputs pre-processed video data into an AI algorithm to detect bicycles. TensorFlow and PyTorch are used for this purpose. The input is pre-processed video data, and the output is identification information indicating the presence of bicycles. Specifically, object detection calculations are performed by the model for each frame.

[0173] Step 3:

[0174] The server compares the location information of identified bicycles with a geographic information database. The PostGIS library is used for this process. The input is location coordinate information obtained through object identification, and the output is the result of determining whether the location is within an illegal parking area. Specifically, a coordinate matching calculation is performed.

[0175] Step 4:

[0176] The server notifies the relevant authorities if it detects illegal parking in a prohibited area. Notifications are sent via email or API. The input is the result of the illegal parking determination, and the output is the logging and transmission of the notification content. Specifically, communication with the mail server takes place.

[0177] Step 5:

[0178] The device captures the user's facial expressions and voice using its camera and microphone and sends them to an emotion engine. This data is analyzed using Google Cloud Vision or IBM Watson. The input is the user's real-time facial expressions and voice, and the output is analyzed emotion information. Specifically, this involves frequency analysis of the voice and feature extraction of facial expressions.

[0179] Step 6:

[0180] The device provides users with appropriate information based on detected emotions. Specifically, it uses the Google Maps API to guide users to nearby bicycle parking spaces. The input is analyzed emotion information, and the output is map information provided to the user. The specific operations include map search and display of results.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0183] Illegal bicycle parking in urban areas obstructs traffic and hinders the efficient use of public spaces. Furthermore, the inability to find parking spaces can cause stress for users. Given these issues, there is a need for an interactive management system that balances public safety with user convenience.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for identifying an object based on the preprocessed video data and determining whether the identified object is located in a predetermined prohibited area, and means for analyzing the user's psychological state from their face and voice and providing interactive guidance based on the analysis results. This enables not only detection and notification of illegal parking, but also the provision of appropriate parking guidance based on the user's emotions, thereby improving public safety and reducing user stress.

[0186] "Acquisition device" refers to hardware or software used to acquire video data of a monitored object.

[0187] "Preprocessing" refers to the process of performing initial processing to convert received video data into a format that allows for object identification.

[0188] "Object identification" refers to recognizing a specific object from pre-processed video data and extracting its characteristics.

[0189] A "designated prohibited area" refers to a specific geographical area where the presence of an object is not permitted.

[0190] "Analyzing a user's psychological state from their face and voice" is a process that evaluates an individual's emotions or psychological tendencies based on their facial expressions and tone of voice.

[0191] "Providing interactive guidance" means dynamically offering information and services that are tailored to the user's psychological state.

[0192] "Notification" refers to the act of transmitting relevant information to a designated institution based on the judgment or analysis results.

[0193] "Relevant organization" refers to the body or organization responsible for responding when it receives notification.

[0194] The server uses OpenCV and other image recognition libraries to preprocess video data acquired from monitoring devices. This allows it to identify objects such as bicycles and determine if they are in a prohibited area. The object's location information is cross-referenced with a Geographic Information System (GIS) database, and if illegal parking is detected, the system automatically notifies the relevant authorities via email or API.

[0195] The device uses the Emotion Recognition API to analyze the user's facial expressions and voice tone. This analysis is then used to provide information about the optimal parking space if the user is experiencing any psychological distress. By interactively receiving information through the device, the stress of parking is reduced.

[0196] For example, when a user is looking for a bicycle parking space while traveling around a train station, the terminal displays the availability of parking spaces via video data. Furthermore, if it senses impatience from the user's facial expression, it provides prompt guidance tailored to their psychological state. This system aims to improve the efficient management of public spaces in cities and enhance user convenience.

[0197] An example of a prompt message when using a generative AI model is, "Please tell me how to suggest the best parking location when the user is feeling stressed."

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The server receives video data transmitted from the acquisition device and performs preprocessing using image recognition libraries such as OpenCV. This process analyzes the video data and converts it into a format suitable for object detection models. The input is raw video data, and the output is preprocessed data suitable for object detection.

[0201] Step 2:

[0202] The server identifies objects (e.g., bicycles) using pre-processed data. By applying an object detection algorithm, it extracts the location information of bicycles within the video. The input is pre-processed data, and the output is data containing the location information of the identified bicycles.

[0203] Step 3:

[0204] The server compares the location information of identified objects with a Geographic Information System (GIS) database and determines whether the objects are located within a restricted area. The input is the location information of the objects, and the output is the result of the area determination.

[0205] Step 4:

[0206] The server notifies the relevant authorities of the presence of an object within a restricted area. This notification is sent automatically via email or an application programming interface (API). The input is the area detection result, and the output is the notification transmission.

[0207] Step 5:

[0208] The device acquires the user's facial and voice data and analyzes their psychological state using the Emotion Recognition API. The input is the user's facial and voice data, and the output is the analyzed emotional information.

[0209] Step 6:

[0210] The device provides interactive guidance to the user based on an analysis of their psychological state. For example, if the user is anxious while searching for a bicycle parking spot, it will present information on the optimal parking location. The input is emotional information, and the output is guidance information for the user.

[0211] Step 7:

[0212] The user adjusts their actions based on guidance from the device. This reduces the stress of choosing a parking spot and moving around. The input is guidance information provided by the device, and the output is the user's actions.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] This invention is a system that analyzes video data from an acquisition device, with the aim of rapidly and effectively detecting and reporting illegal parking. The system utilizes AI-powered object recognition technology and a database-based method for determining prohibited areas.

[0230] The server first receives video data from the acquisition device and performs preprocessing on that data. This preprocessing includes noise reduction and image resizing. Next, the server passes the preprocessed data to an AI algorithm for object recognition. This determines where objects such as bicycles are located within the video.

[0231] Subsequently, the server checks the location information of the identified bicycle against a database of prohibited areas to determine if it is illegally parked. Once this determination is made, the server records the information of the illegally parked bicycle and initiates the process of notifying the relevant authorities.

[0232] As a concrete example, imagine a camera installed in a busy area of ​​a city that continuously captures video in real time. The server uses AI to detect bicycles parked at specific locations from this video, and if they are within a designated prohibited area, it automatically records that fact. The server then notifies the police or city hall via email to prompt action against illegal parking.

[0233] The terminal provides an interface that allows users to check information on illegally parked bicycles in real time, helping the general public to park their bicycles responsibly and with awareness.

[0234] Thus, this system is effective in addressing the problem of illegal bicycle parking in urban areas and contributes to improving public safety and comfort.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The server receives video data in real time from the acquisition device. This video data is transmitted from cameras designed to continuously cover the area being monitored.

[0238] Step 2:

[0239] The server performs preprocessing on the received video data. Specifically, it removes noise from the video and resizes the resolution to a size suitable for the AI ​​algorithm.

[0240] Step 3:

[0241] The server inputs the pre-processed video into an AI algorithm and begins object recognition. Here, a convolutional neural network (CNN) is used to detect bicycles in the image and determine their location.

[0242] Step 4:

[0243] The server then compares the location information of the identified bicycle with a geographic information database. This allows it to determine whether the identified bicycle is located within a no-parking zone.

[0244] Step 5:

[0245] The server records information in its database if it determines that a bicycle is illegally parked. This record includes the date and time of detection, location information, and a video snapshot.

[0246] Step 6:

[0247] The server automatically generates a notification message based on recorded illegal parking information and sends the notification to the relevant authorities in the appropriate format. This notification is sent via email or a dedicated API.

[0248] Step 7:

[0249] The terminal provides a user interface that allows users to view information on illegally parked bicycles. This enables users to understand the situation of illegal parking in real time and to be more mindful of preventing parking problems.

[0250] (Example 1)

[0251] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0252] Illegal parking of bicycles poses a significant problem for urban traffic, landscape, and crime prevention. Conventional measures require substantial human resources for monitoring and enforcement, making effective control difficult. This invention aims to provide a system that efficiently and quickly detects illegally parked bicycles with limited resources and notifies the relevant authorities.

[0253] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0254] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for performing object identification using a generated AI model with the preprocessed video data, determining the location of the identified object, and determining whether the location matches a predetermined prohibited area database, and means for notifying relevant organizations of information about the identified object based on the determination result via electronic communication means. This makes it possible to automatically detect illegal parking and notify relevant parties quickly and with high accuracy.

[0255] An "acquisition device" is a device used to acquire video data from an object being observed, and includes devices such as cameras and sensors.

[0256] "Preprocessing" refers to the initial processing of received video data, such as noise reduction and resizing.

[0257] A "generative AI model" is a model that uses machine learning algorithms and is a mathematical representation for performing data analysis or object recognition in specific tasks.

[0258] "Object recognition" refers to the technology of detecting and classifying specific objects contained within image data.

[0259] A "restricted area database" is a database that holds information about restrictions in a specific area, including information about geographical boundaries.

[0260] "Electronic communication methods" refer to technologies and protocols used for digital communication via email or networks.

[0261] "Geographic information management technology" refers to technologies for collecting, managing, and analyzing geographic information, and includes GIS (Geographic Information Systems).

[0262] In implementing this invention, the server is primarily responsible for data reception, processing, and notification. The server receives video data in real time from cameras installed throughout the city as acquisition devices. These cameras have appropriate image quality and frame rates, enabling surveillance at various points in the city.

[0263] The server preprocesses the received video data using image processing libraries such as OpenCV. This preprocessing includes noise reduction, image quality improvement, and resolution modification. This processing improves the accuracy of object recognition by the subsequent generative AI model.

[0264] Next, the server applies machine learning algorithms such as YOLO (You Only Look Once) as a "generative AI model." This model quickly and accurately identifies objects such as bicycles present in the video. Then, by extracting the location data of the identified objects, it identifies where illegal parking is occurring.

[0265] The server uses geographic information management technologies such as GIS to compare identified location data with a database of prohibited areas. This database includes prohibited areas pre-defined based on urban planning and transportation policies. This allows the server to immediately determine whether or not illegal parking is present.

[0266] If the assessment confirms illegal parking, the server will use electronic communication methods to send a notification to the relevant authorities via email or a dedicated application. This notification will include the location information of the illegally parked bicycle and related image data, enabling a swift response.

[0267] Furthermore, the terminal provides an interface that allows users to check information on illegally parked bicycles in real time. This terminal enables users to understand the current state of illegal parking and contribute to improving parking etiquette.

[0268] A concrete example would be a process where cameras installed in the city center continuously transmit video footage to a server, which then analyzes the data to immediately identify illegal parking at specific locations and notifies the police and public transport authorities.

[0269] An example of a prompt is, "Explain the procedure for detecting objects from real-time video and comparing them to prohibited areas," which demonstrates an application using a generative AI model.

[0270] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0271] Step 1:

[0272] The server receives video data from the acquisition device. The input is real-time video data from the camera, and by acquiring this data, the server is ready to begin detecting illegally parked bicycles. Specifically, it continuously receives digital video signals via the network.

[0273] Step 2:

[0274] The server performs preprocessing on the received video data. The input is the received video data, and the output is image data with noise removed and the resolution appropriately adjusted. Here, the OpenCV library is used to perform noise reduction and image resizing, thereby preparing the data for subsequent analysis.

[0275] Step 3:

[0276] The server inputs the preprocessed image data into the generative AI model for object identification. The input is the preprocessed image data, and the output is the data specifying the position of the bicycle in the image. Specifically, the YOLO model is applied to detect the bicycle in each frame and extract its coordinate information.

[0277] Step 4:

[0278] The server uses geospatial information management technology with the identified object's position information and compares it with the no-go area database. The input is the identified position information, and the output is the determination result of whether the object is within the no-go area. In this process, the position information is cross-checked by GIS to determine the possibility of illegal parking.

[0279] Step 5:

[0280] Based on the determination result, the server notifies the relevant agency of the information on the bicycles determined to be illegal. The input is the matching result with the no-go area, and the output is the detailed information on the notified illegal parking. Specifically, it automatically reports to the necessary agencies using email or API and transmits the illegal parking information including the image data.

[0281] Step 6:

[0282] The terminal provides the user with real-time illegal parking information. The input is the notification information received from the server, and the output is the situation of illegal parking displayed on the user interface. In this process, the information is presented in a visually understandable form so that the user can confirm it immediately.

[0283] (Application Example 1)

[0284] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0285] The problem of illegal parking in urban areas is a major issue in urban life, such as obstructing traffic, damaging aesthetics, and hindering evacuation routes in case of emergencies. However, due to a shortage of manpower and limitations in the monitoring scope in the current monitoring system, it is difficult to quickly detect and respond to illegal parking. In addition, there is a lack of guidelines for citizens to choose appropriate parking locations. Therefore, there is a need to establish a system that enables real-time detection and reporting of illegal parking and guidance on parking locations for citizens.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0287] In this invention, the server includes means for preprocessing the visual data of the monitoring target received from the acquisition device, means for identifying an object based on the preprocessed visual data and determining whether the identified object exists in a predetermined prohibited area, means for notifying the relevant agency of the information of the identified object based on the determination result, and means for notifying the digital terminal in real time of the information of the object within the predetermined prohibited area to the user and guiding an appropriate parking location. Thereby, it becomes possible to quickly detect and respond to illegal parking and guide an appropriate parking location to citizens.

[0288] The "acquisition device" is hardware or a system for acquiring visual data of the monitoring target.

[0289] "Preprocessing" is a process of performing noise removal and size adjustment on the acquired visual data and converting it into a state suitable for analysis.

[0290] "Object identification" is a technology for detecting a specific object existing in visual data by an algorithm such as AI and confirming what it is.

[0291] The "prohibited area" is information indicating a specific area where the presence of an object is not permitted, and it is collated with a geographical information database.

[0292] "Related organizations" refer to organizations or entities that receive information about illegal parking and take necessary action.

[0293] "Notification" is the act of transmitting information about an identified object to a designated recipient using electronic means of communication.

[0294] A "digital device" is an electronic device that provides a user interface, and includes smartphones and tablets.

[0295] "Real-time" refers to the ability to process or provide data instantly, without delay.

[0296] "Bicycle parking location guidance" is a service that provides users with information and guidance on appropriate bicycle parking locations.

[0297] This invention comprises a system for detecting and notifying illegal bicycle parking using a digital terminal. The server receives visual data from acquisition devices installed in the monitored area and performs preprocessing. This preprocessing includes noise reduction and image resizing. As a result, the data is appropriately analyzed by an AI algorithm, enabling object identification.

[0298] The server uses TensorFlow or similar methods to perform object recognition and detect objects such as bicycles in the visual data. The identified objects are then compared against prohibited area information stored in a geographic information database, and if a match is found, it is determined to be illegally parked. Information regarding illegal parking is immediately notified to the relevant authorities via electronic communication.

[0299] At the same time, users with digital devices can receive notifications in real time. The devices display on-site information about illegally parked bicycles and also provide guidance to appropriate parking locations. For example, if a citizen tries to park their bicycle near the entrance of a park, the device can issue an alert saying, "This is a no-parking zone," and display information about the nearest parking area along with a map.

[0300] Through the user experience, it aims to achieve rapid response to illegal parking in urban areas and appropriate parking guidance. In the construction of the system according to this invention, object identification using a generative AI model and an instant communication function play important roles.

[0301] Examples of input prompt sentences for the generative AI model for specific scenarios are as follows.

[0302] "Please explain the method of identifying bicycles parked in a prohibited area from camera images and generating an alert message when detected."

[0303] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0304] Step 1:

[0305] The server receives visual input data from the acquisition device. The received data is acquired as an unprocessed video stream. This data is used as input for processing in the next step.

[0306] Step 2:

[0307] The server performs preprocessing on the received visual data. The preprocessing includes operations such as removing noise from the image and resizing the image to a size suitable for handling by the AI model. Appropriately formatted video data is output as the preprocessing result.

[0308] Step 3:

[0309] The server analyzes the preprocessed video data using a generative AI model to perform object identification. Specifically, the AI model detects specific objects such as bicycles in the video. The obtained identification information is generated as output and used in subsequent steps.

[0310] Step 4:

[0311] The server compares the location information of the identified object with a geographic information database. This comparison determines whether the object is located within a no-parking zone. The result of the determination (whether or not illegal parking is present) and the object's location information are output.

[0312] Step 5:

[0313] If illegal parking is detected, the server notifies the relevant authorities of the determination. The notification is made automatically using electronic communication methods, and the content of the report is output.

[0314] Step 6:

[0315] The device notifies the user in real time about objects within the restricted area. The notification is sent to the device, and guidance to a suitable parking location is also provided. This notification allows the user to immediately understand the current situation in the restricted area and take appropriate action.

[0316] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0317] This invention provides a more interactive and comprehensive management system by combining an illegal parking detection and notification system with an emotion engine that recognizes user emotions. This system operates by integrating object recognition capabilities using video data with emotion analysis.

[0318] The server preprocesses the video data received from the acquisition device and uses an AI algorithm to detect bicycles. The server then compares the location of the detected bicycles with a geographic information database to determine if they are illegally parked in a prohibited area. If illegal parking is confirmed, the server notifies the relevant authorities based on the recorded information.

[0319] This system also features an emotion engine that identifies emotions from the user's facial expressions and voice. When a user interacts with the system, the terminal analyzes the user's emotions in real time and records them in a database. For example, if a user looking for a place to park their bicycle is feeling stressed, the emotion engine can identify that emotion and provide information about available parking locations.

[0320] As a concrete example, consider a scenario where a user is using a device to search for a bicycle parking space. In this case, the system analyzes the user's voice and facial expressions using an emotion engine. If it detects anxiety or impatience, the device displays a map and guides the user to nearby available parking spaces. This information can also be shared with relevant organizations for safety management purposes.

[0321] In this way, this system, which incorporates an emotion engine, achieves both a better user experience and improved public safety by integrating object detection and emotion analysis.

[0322] The following describes the processing flow.

[0323] Step 1:

[0324] The server receives video data in real time from the acquisition device. This data captures the situation within the monitoring area.

[0325] Step 2:

[0326] The server performs preprocessing on the received video data. Specifically, it removes noise from the video and adjusts the image to a size that is easy to analyze.

[0327] Step 3:

[0328] The server inputs the pre-processed video data into an AI algorithm to perform object recognition. Here, it detects bicycles and other objects and determines their location.

[0329] Step 4:

[0330] The server then compares the location information of the identified object with a geographic information database to determine whether it is within a restricted area.

[0331] Step 5:

[0332] The server notifies the relevant authorities of details about objects determined to be illegally parked. This notification is made promptly via email or API.

[0333] Step 6:

[0334] The device captures the user's facial expressions and voice and sends them to the emotion engine. This is to analyze the user's emotions in real time while they are interacting with the system.

[0335] Step 7:

[0336] The server analyzes emotional data obtained from the emotion engine and generates feedback tailored to the user's state. For example, if the user is feeling stressed, it will present supportive information.

[0337] Step 8:

[0338] The device displays necessary information to the user based on emotional data. This includes providing specific guidance on areas where bicycles can be parked, as well as messages aimed at addressing the user's emotional needs.

[0339] Through this process, the system can not only manage illegal parking but also respond to users' emotions, aiming to maintain a more interactive and effective urban environment.

[0340] (Example 2)

[0341] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0342] In modern urban environments, illegal bicycle parking is a significant problem that severely impairs public traffic and aesthetics. However, simply monitoring parking is not enough; responses that consider the feelings of users are also required. Therefore, in addition to detecting illegal parking, it is necessary to understand the feelings of users and provide appropriate information accordingly. A comprehensive system is needed to address these complex issues.

[0343] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0344] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for identifying an object based on the preprocessed video data and determining whether the identified object is located in a predetermined prohibited area, and means for analyzing the user's facial expressions and voice to identify their emotions. This enables the detection of illegal parking and the provision of information based on the user's emotions.

[0345] An "acquisition device" is a device that acquires video data from the area being monitored.

[0346] "Video data" refers to data that digitally records the visual information of the subject being monitored.

[0347] "Preprocessing" refers to the initial stage of data processing performed to make video data easier to analyze.

[0348] "Object" refers to an identifiable entity within video data, and in this invention, it primarily refers to a bicycle.

[0349] "Identification" refers to the act of identifying the type and characteristics of an object based on video data.

[0350] A "forbidden area" refers to a designated location where the presence of objects is not permitted.

[0351] "Determination" is the act of determining whether or not an identified object exists in a prohibited area.

[0352] "Relevant body" refers to any organization or body that has the authority to take appropriate action upon receiving notification.

[0353] "Notification" refers to the act of transmitting information, such as judgment results, to other organizations or devices.

[0354] "Facial expressions and voice" refer to the user's visual and auditory expressions of emotion.

[0355] "Emotions" are elements that indicate the user's psychological state and are identified by the system.

[0356] "Information provision" refers to the act of presenting useful data to users.

[0357] This invention integrates a module that analyzes user emotions into a system for detecting and reporting illegally parked bicycles. The system incorporates both object detection and emotion recognition capabilities, aiming to improve the user experience and maintain public order.

[0358] The server receives video data from acquisition devices such as surveillance cameras. This data is preprocessed using the OpenCV library, including noise reduction and resolution adjustment. The preprocessed data is then analyzed by AI algorithms utilizing TensorFlow, PyTorch, etc., to detect bicycles in the video. The location information of the detected objects is cross-referenced with a geographic information database to evaluate whether they fall within a prohibited area. Database tools such as PostGIS are used for this process. If parking in a prohibited area is confirmed, the server automatically notifies the relevant authorities. The notification is securely transmitted via email or application programming interface.

[0359] On the other hand, when a user interacts with the device, the device uses its camera and microphone to capture the user's facial expressions and voice in real time. By analyzing this data using sentiment analysis APIs such as Google Cloud Vision and IBM Watson, the system identifies the user's emotions. For example, if the system detects impatience or anxiety while a user is looking for a bicycle parking space, the device uses the Google Maps API to guide the user to the nearest available parking space.

[0360] As a concrete example, consider a situation where a user is looking for a bicycle parking space near a train station. The system detects the user's voice and facial expressions, and if it detects "anxiety," the terminal displays "We will guide you to a nearby bicycle parking space" on the screen and provides a function to suggest the nearest possible locations. In this case, the system will respond if the user says, "Tell me about nearby bicycle parking spaces."

[0361] This configuration allows the system to check for parking violations in real time while providing services tailored to the user's emotions, creating a more efficient and user-friendly environment.

[0362] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0363] Step 1:

[0364] The server receives video data in real time from the acquisition device. This data is preprocessed using the OpenCV library. The input is video data from the surveillance camera, and the output is high-quality video data with noise reduction. Specifically, filtering is performed to reduce shadows and noise.

[0365] Step 2:

[0366] The server inputs pre-processed video data into an AI algorithm to detect bicycles. TensorFlow and PyTorch are used for this purpose. The input is pre-processed video data, and the output is identification information indicating the presence of bicycles. Specifically, object detection calculations are performed by the model for each frame.

[0367] Step 3:

[0368] The server compares the location information of identified bicycles with a geographic information database. The PostGIS library is used for this process. The input is location coordinate information obtained through object identification, and the output is the result of determining whether the location is within an illegal parking area. Specifically, a coordinate matching calculation is performed.

[0369] Step 4:

[0370] The server notifies the relevant authorities if it detects illegal parking in a prohibited area. Notifications are sent via email or API. The input is the result of the illegal parking determination, and the output is the logging and transmission of the notification content. Specifically, communication with the mail server takes place.

[0371] Step 5:

[0372] The device captures the user's facial expressions and voice using its camera and microphone and sends them to an emotion engine. This data is analyzed using Google Cloud Vision or IBM Watson. The input is the user's real-time facial expressions and voice, and the output is analyzed emotion information. Specifically, this involves frequency analysis of the voice and feature extraction of facial expressions.

[0373] Step 6:

[0374] The device provides users with appropriate information based on detected emotions. Specifically, it uses the Google Maps API to guide users to nearby bicycle parking spaces. The input is analyzed emotion information, and the output is map information provided to the user. The specific operations include map search and display of results.

[0375] (Application Example 2)

[0376] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0377] Illegal bicycle parking in urban areas obstructs traffic and hinders the efficient use of public spaces. Furthermore, the inability to find parking spaces can cause stress for users. Given these issues, there is a need for an interactive management system that balances public safety with user convenience.

[0378] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0379] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for identifying an object based on the preprocessed video data and determining whether the identified object is located in a predetermined prohibited area, and means for analyzing the user's psychological state from their face and voice and providing interactive guidance based on the analysis results. This enables not only detection and notification of illegal parking, but also the provision of appropriate parking guidance based on the user's emotions, thereby improving public safety and reducing user stress.

[0380] "Acquisition device" refers to hardware or software used to acquire video data of a monitored object.

[0381] "Preprocessing" refers to the process of performing initial processing to convert received video data into a format that allows for object identification.

[0382] "Object identification" refers to recognizing a specific object from pre-processed video data and extracting its characteristics.

[0383] A "designated prohibited area" refers to a specific geographical area where the presence of an object is not permitted.

[0384] "Analyzing a user's psychological state from their face and voice" is a process that evaluates an individual's emotions or psychological tendencies based on their facial expressions and tone of voice.

[0385] "Providing interactive guidance" means dynamically offering information and services that are tailored to the user's psychological state.

[0386] "Notification" refers to the act of transmitting relevant information to a designated institution based on the judgment or analysis results.

[0387] "Relevant organization" refers to the body or organization responsible for responding when it receives notification.

[0388] The server uses OpenCV and other image recognition libraries to preprocess video data acquired from monitoring devices. This allows it to identify objects such as bicycles and determine if they are in a prohibited area. The object's location information is cross-referenced with a Geographic Information System (GIS) database, and if illegal parking is detected, the system automatically notifies the relevant authorities via email or API.

[0389] The device uses the Emotion Recognition API to analyze the user's facial expressions and voice tone. This analysis is then used to provide information about the optimal parking space if the user is experiencing any psychological distress. By interactively receiving information through the device, the stress of parking is reduced.

[0390] For example, when a user is looking for a bicycle parking space while traveling around a train station, the terminal displays the availability of parking spaces via video data. Furthermore, if it senses impatience from the user's facial expression, it provides prompt guidance tailored to their psychological state. This system aims to improve the efficient management of public spaces in cities and enhance user convenience.

[0391] An example of a prompt message when using a generative AI model is, "Please tell me how to suggest the best parking location when the user is feeling stressed."

[0392] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0393] Step 1:

[0394] The server receives video data transmitted from the acquisition device and performs preprocessing using image recognition libraries such as OpenCV. This process analyzes the video data and converts it into a format suitable for object detection models. The input is raw video data, and the output is preprocessed data suitable for object detection.

[0395] Step 2:

[0396] The server identifies objects (e.g., bicycles) using pre-processed data. By applying an object detection algorithm, it extracts the location information of bicycles within the video. The input is pre-processed data, and the output is data containing the location information of the identified bicycles.

[0397] Step 3:

[0398] The server compares the location information of identified objects with a Geographic Information System (GIS) database and determines whether the objects are located within a restricted area. The input is the location information of the objects, and the output is the result of the area determination.

[0399] Step 4:

[0400] The server notifies the relevant authorities of the presence of an object within a restricted area. This notification is sent automatically via email or an application programming interface (API). The input is the area detection result, and the output is the notification transmission.

[0401] Step 5:

[0402] The device acquires the user's facial and voice data and analyzes their psychological state using the Emotion Recognition API. The input is the user's facial and voice data, and the output is the analyzed emotional information.

[0403] Step 6:

[0404] The device provides interactive guidance to the user based on an analysis of their psychological state. For example, if the user is anxious while searching for a bicycle parking spot, it will present information on the optimal parking location. The input is emotional information, and the output is guidance information for the user.

[0405] Step 7:

[0406] The user adjusts their actions based on guidance from the device. This reduces the stress of choosing a parking spot and moving around. The input is guidance information provided by the device, and the output is the user's actions.

[0407] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0408] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0409] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0410] [Third Embodiment]

[0411] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0412] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0413] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0414] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0415] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0416] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0417] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0418] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0419] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0420] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0421] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0422] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0423] This invention is a system that analyzes video data from an acquisition device, with the aim of rapidly and effectively detecting and reporting illegal parking. The system utilizes AI-powered object recognition technology and a database-based method for determining prohibited areas.

[0424] The server first receives video data from the acquisition device and performs preprocessing on that data. This preprocessing includes noise reduction and image resizing. Next, the server passes the preprocessed data to an AI algorithm for object recognition. This determines where objects such as bicycles are located within the video.

[0425] Subsequently, the server checks the location information of the identified bicycle against a database of prohibited areas to determine if it is illegally parked. Once this determination is made, the server records the information of the illegally parked bicycle and initiates the process of notifying the relevant authorities.

[0426] As a concrete example, imagine a camera installed in a busy area of ​​a city that continuously captures video in real time. The server uses AI to detect bicycles parked at specific locations from this video, and if they are within a designated prohibited area, it automatically records that fact. The server then notifies the police or city hall via email to prompt action against illegal parking.

[0427] The terminal provides an interface that allows users to check information on illegally parked bicycles in real time, helping the general public to park their bicycles responsibly and with awareness.

[0428] Thus, this system is effective in addressing the problem of illegal bicycle parking in urban areas and contributes to improving public safety and comfort.

[0429] The following describes the processing flow.

[0430] Step 1:

[0431] The server receives video data in real time from the acquisition device. This video data is transmitted from cameras designed to continuously cover the area being monitored.

[0432] Step 2:

[0433] The server performs preprocessing on the received video data. Specifically, it removes noise from the video and resizes the resolution to a size suitable for the AI ​​algorithm.

[0434] Step 3:

[0435] The server inputs the pre-processed video into an AI algorithm and begins object recognition. Here, a convolutional neural network (CNN) is used to detect bicycles in the image and determine their location.

[0436] Step 4:

[0437] The server then compares the location information of the identified bicycle with a geographic information database. This allows it to determine whether the identified bicycle is located within a no-parking zone.

[0438] Step 5:

[0439] The server records information in its database if it determines that a bicycle is illegally parked. This record includes the date and time of detection, location information, and a video snapshot.

[0440] Step 6:

[0441] The server automatically generates a notification message based on recorded illegal parking information and sends the notification to the relevant authorities in the appropriate format. This notification is sent via email or a dedicated API.

[0442] Step 7:

[0443] The terminal provides a user interface that allows users to view information on illegally parked bicycles. This enables users to understand the situation of illegal parking in real time and to be more mindful of preventing parking problems.

[0444] (Example 1)

[0445] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0446] Illegal parking of bicycles poses a significant problem for urban traffic, landscape, and crime prevention. Conventional measures require substantial human resources for monitoring and enforcement, making effective control difficult. This invention aims to provide a system that efficiently and quickly detects illegally parked bicycles with limited resources and notifies the relevant authorities.

[0447] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0448] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for performing object identification using a generated AI model with the preprocessed video data, determining the location of the identified object, and determining whether the location matches a predetermined prohibited area database, and means for notifying relevant organizations of information about the identified object based on the determination result via electronic communication means. This makes it possible to automatically detect illegal parking and notify relevant parties quickly and with high accuracy.

[0449] An "acquisition device" is a device used to acquire video data from an object being observed, and includes devices such as cameras and sensors.

[0450] "Preprocessing" refers to the initial processing of received video data, such as noise reduction and resizing.

[0451] A "generative AI model" is a model that uses machine learning algorithms and is a mathematical representation for performing data analysis or object recognition in specific tasks.

[0452] "Object recognition" refers to the technology of detecting and classifying specific objects contained within image data.

[0453] A "restricted area database" is a database that holds information about restrictions in a specific area, including information about geographical boundaries.

[0454] "Electronic communication methods" refer to technologies and protocols used for digital communication via email or networks.

[0455] "Geographic information management technology" refers to technologies for collecting, managing, and analyzing geographic information, and includes GIS (Geographic Information Systems).

[0456] In implementing this invention, the server is primarily responsible for data reception, processing, and notification. The server receives video data in real time from cameras installed throughout the city as acquisition devices. These cameras have appropriate image quality and frame rates, enabling surveillance at various points in the city.

[0457] The server preprocesses the received video data using image processing libraries such as OpenCV. This preprocessing includes noise reduction, image quality improvement, and resolution modification. This processing improves the accuracy of object recognition by the subsequent generative AI model.

[0458] Next, the server applies machine learning algorithms such as YOLO (You Only Look Once) as a "generative AI model." This model quickly and accurately identifies objects such as bicycles present in the video. Then, by extracting the location data of the identified objects, it identifies where illegal parking is occurring.

[0459] The server uses geographic information management technologies such as GIS to compare identified location data with a database of prohibited areas. This database includes prohibited areas pre-defined based on urban planning and transportation policies. This allows the server to immediately determine whether or not illegal parking is present.

[0460] If the assessment confirms illegal parking, the server will use electronic communication methods to send a notification to the relevant authorities via email or a dedicated application. This notification will include the location information of the illegally parked bicycle and related image data, enabling a swift response.

[0461] Furthermore, the terminal provides an interface that allows users to check information on illegally parked bicycles in real time. This terminal enables users to understand the current state of illegal parking and contribute to improving parking etiquette.

[0462] A concrete example would be a process where cameras installed in the city center continuously transmit video footage to a server, which then analyzes the data to immediately identify illegal parking at specific locations and notifies the police and public transport authorities.

[0463] An example of a prompt is, "Explain the procedure for detecting objects from real-time video and comparing them to prohibited areas," which demonstrates an application using a generative AI model.

[0464] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0465] Step 1:

[0466] The server receives video data from the acquisition device. The input is real-time video data from the camera, and by acquiring this data, the server is ready to begin detecting illegally parked bicycles. Specifically, it continuously receives digital video signals via the network.

[0467] Step 2:

[0468] The server performs preprocessing on the received video data. The input is the received video data, and the output is image data with noise removed and the resolution appropriately adjusted. Here, the OpenCV library is used to perform noise reduction and image resizing, thereby preparing the data for subsequent analysis.

[0469] Step 3:

[0470] The server inputs preprocessed image data into a generating AI model to perform object recognition. The input is preprocessed image data, and the output is data identifying the position of bicycles within the image. Specifically, the YOLO model is applied to detect bicycles in each frame and extract their coordinate information.

[0471] Step 4:

[0472] The server uses geographic information management technology (GIS) to compare the location information of identified objects against a prohibited area database. The input is the identified location information, and the output is the result of determining whether the object is within a prohibited area. In this process, the location information is cross-referenced using GIS to determine the possibility of illegal parking.

[0473] Step 5:

[0474] Based on the judgment results, the server notifies the relevant authorities of information about bicycles deemed illegally parked. The input is the result of matching against prohibited areas, and the output is detailed information about the reported illegally parked bicycles. Specifically, it automatically notifies the necessary authorities via email or API and transmits illegal parking information, including image data.

[0475] Step 6:

[0476] The terminal provides users with real-time information on illegally parked bicycles. Input is the reported information received from the server, and output is the illegal parking status displayed on the user interface. This process presents information in a visually easy-to-understand format, allowing users to check it immediately.

[0477] (Application Example 1)

[0478] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0479] Illegal bicycle parking in urban areas is a major problem for city life, as it obstructs traffic, damages the aesthetics of the area, and hinders emergency evacuation routes. However, the current monitoring system suffers from a shortage of personnel and limitations in monitoring area, making it difficult to quickly detect and respond to illegal parking. Furthermore, there is a lack of guidance for citizens to choose appropriate parking locations. Therefore, there is a need to establish a system that enables real-time detection and reporting of illegal parking, as well as guidance to citizens on suitable parking locations.

[0480] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0481] In this invention, the server includes means for preprocessing visual data of a monitored object received from an acquisition device; means for identifying an object based on the preprocessed visual data and determining whether the identified object is located in a predetermined prohibited area; means for notifying relevant organizations of information about the identified object based on the determination result; and means for notifying users of a digital terminal in real time about objects located within the predetermined prohibited area and guiding them to an appropriate parking location. This enables rapid detection and response to illegal parking and guidance of citizens to appropriate parking locations.

[0482] An "acquisition device" is hardware or a system used to acquire visual data of a monitored object.

[0483] "Preprocessing" refers to the process of removing noise and adjusting the size of acquired visual data to convert it into a state suitable for analysis.

[0484] "Object recognition" is a technology that uses algorithms such as AI to detect specific objects present in visual data and to identify what they are.

[0485] A "restricted area" is information indicating a specific area where the presence of objects is not permitted, and this information is cross-referenced with a geographic information database.

[0486] "Related organizations" refer to organizations or entities that receive information about illegal parking and take necessary action.

[0487] "Notification" is the act of transmitting information about an identified object to a designated recipient using electronic means of communication.

[0488] A "digital device" is an electronic device that provides a user interface, and includes smartphones and tablets.

[0489] "Real-time" refers to the ability to process or provide data instantly, without delay.

[0490] "Bicycle parking location guidance" is a service that provides users with information and guidance on appropriate bicycle parking locations.

[0491] This invention comprises a system for detecting and notifying illegal bicycle parking using a digital terminal. The server receives visual data from acquisition devices installed in the monitored area and performs preprocessing. This preprocessing includes noise reduction and image resizing. As a result, the data is appropriately analyzed by an AI algorithm, enabling object identification.

[0492] The server uses TensorFlow or similar methods to perform object recognition and detect objects such as bicycles in the visual data. The identified objects are then compared against prohibited area information stored in a geographic information database, and if a match is found, it is determined to be illegally parked. Information regarding illegal parking is immediately notified to the relevant authorities via electronic communication.

[0493] At the same time, users with digital devices can receive notifications in real time. The devices display on-site information about illegally parked bicycles and also provide guidance to appropriate parking locations. For example, if a citizen tries to park their bicycle near the entrance of a park, the device can issue an alert saying, "This is a no-parking zone," and display information about the nearest parking area along with a map.

[0494] The aim is to enable rapid response to illegal bicycle parking in urban areas and appropriate parking guidance through the user experience. In the system construction based on this invention, object recognition using a generative AI model and instant communication functions play important roles.

[0495] Examples of input prompts for a generative AI model for specific scenarios are as follows:

[0496] "Please explain how to identify bicycles parked in restricted areas using camera footage and generate an alert message when they are detected."

[0497] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0498] Step 1:

[0499] The server receives visual input data from the acquisition device. The received data is acquired as a raw video stream. This data is used as input for processing in the next step.

[0500] Step 2:

[0501] The server performs preprocessing on the received visual data. This preprocessing includes removing noise from the image and resizing it to a size that is easily handled by the AI ​​model. As a result of the preprocessing, appropriately formatted video data is output.

[0502] Step 3:

[0503] The server analyzes the pre-processed video data using an AI model to perform object identification. Specifically, the AI ​​model detects specific objects in the video, such as bicycles. The obtained identification information is generated as output and used in subsequent steps.

[0504] Step 4:

[0505] The server compares the location information of the identified object with a geographic information database. This comparison determines whether the object is located within a no-parking zone. The result of the determination (whether or not illegal parking is present) and the object's location information are output.

[0506] Step 5:

[0507] If illegal parking is detected, the server notifies the relevant authorities of the determination. The notification is made automatically using electronic communication methods, and the content of the report is output.

[0508] Step 6:

[0509] The device notifies the user in real time about objects within the restricted area. The notification is sent to the device, and guidance to a suitable parking location is also provided. This notification allows the user to immediately understand the current situation in the restricted area and take appropriate action.

[0510] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0511] This invention provides a more interactive and comprehensive management system by combining an illegal parking detection and notification system with an emotion engine that recognizes user emotions. This system operates by integrating object recognition capabilities using video data with emotion analysis.

[0512] The server preprocesses the video data received from the acquisition device and uses an AI algorithm to detect bicycles. The server then compares the location of the detected bicycles with a geographic information database to determine if they are illegally parked in a prohibited area. If illegal parking is confirmed, the server notifies the relevant authorities based on the recorded information.

[0513] This system also features an emotion engine that identifies emotions from the user's facial expressions and voice. When a user interacts with the system, the terminal analyzes the user's emotions in real time and records them in a database. For example, if a user looking for a place to park their bicycle is feeling stressed, the emotion engine can identify that emotion and provide information about available parking locations.

[0514] As a concrete example, consider a scenario where a user is using a device to search for a bicycle parking space. In this case, the system analyzes the user's voice and facial expressions using an emotion engine. If it detects anxiety or impatience, the device displays a map and guides the user to nearby available parking spaces. This information can also be shared with relevant organizations for safety management purposes.

[0515] In this way, this system, which incorporates an emotion engine, achieves both a better user experience and improved public safety by integrating object detection and emotion analysis.

[0516] The following describes the processing flow.

[0517] Step 1:

[0518] The server receives video data in real time from the acquisition device. This data captures the situation within the monitoring area.

[0519] Step 2:

[0520] The server performs preprocessing on the received video data. Specifically, it removes noise from the video and adjusts the image to a size that is easy to analyze.

[0521] Step 3:

[0522] The server inputs the pre-processed video data into an AI algorithm to perform object recognition. Here, it detects bicycles and other objects and determines their location.

[0523] Step 4:

[0524] The server then compares the location information of the identified object with a geographic information database to determine whether it is within a restricted area.

[0525] Step 5:

[0526] The server notifies the relevant authorities of details about objects determined to be illegally parked. This notification is made promptly via email or API.

[0527] Step 6:

[0528] The device captures the user's facial expressions and voice and sends them to the emotion engine. This is to analyze the user's emotions in real time while they are interacting with the system.

[0529] Step 7:

[0530] The server analyzes emotional data obtained from the emotion engine and generates feedback tailored to the user's state. For example, if the user is feeling stressed, it will present supportive information.

[0531] Step 8:

[0532] The device displays necessary information to the user based on emotional data. This includes providing specific guidance on areas where bicycles can be parked, as well as messages aimed at addressing the user's emotional needs.

[0533] Through this process, the system can not only manage illegal parking but also respond to users' emotions, aiming to maintain a more interactive and effective urban environment.

[0534] (Example 2)

[0535] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0536] In modern urban environments, illegal bicycle parking is a significant problem that severely impairs public traffic and aesthetics. However, simply monitoring parking is not enough; responses that consider the feelings of users are also required. Therefore, in addition to detecting illegal parking, it is necessary to understand the feelings of users and provide appropriate information accordingly. A comprehensive system is needed to address these complex issues.

[0537] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0538] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for identifying an object based on the preprocessed video data and determining whether the identified object is located in a predetermined prohibited area, and means for analyzing the user's facial expressions and voice to identify their emotions. This enables the detection of illegal parking and the provision of information based on the user's emotions.

[0539] An "acquisition device" is a device that acquires video data from the area being monitored.

[0540] "Video data" refers to data that digitally records the visual information of the subject being monitored.

[0541] "Preprocessing" refers to the initial stage of data processing performed to make video data easier to analyze.

[0542] "Object" refers to an identifiable entity within video data, and in this invention, it primarily refers to a bicycle.

[0543] "Identification" refers to the act of identifying the type and characteristics of an object based on video data.

[0544] A "forbidden area" refers to a designated location where the presence of objects is not permitted.

[0545] "Determination" is the act of determining whether or not an identified object exists in a prohibited area.

[0546] "Relevant body" refers to any organization or body that has the authority to take appropriate action upon receiving notification.

[0547] "Notification" refers to the act of transmitting information, such as judgment results, to other organizations or devices.

[0548] "Facial expressions and voice" refer to the user's visual and auditory expressions of emotion.

[0549] "Emotions" are elements that indicate the user's psychological state and are identified by the system.

[0550] "Information provision" refers to the act of presenting useful data to users.

[0551] This invention integrates a module that analyzes user emotions into a system for detecting and reporting illegally parked bicycles. The system incorporates both object detection and emotion recognition capabilities, aiming to improve the user experience and maintain public order.

[0552] The server receives video data from acquisition devices such as surveillance cameras. This data is preprocessed using the OpenCV library, including noise reduction and resolution adjustment. The preprocessed data is then analyzed by AI algorithms utilizing TensorFlow, PyTorch, etc., to detect bicycles in the video. The location information of the detected objects is cross-referenced with a geographic information database to evaluate whether they fall within a prohibited area. Database tools such as PostGIS are used for this process. If parking in a prohibited area is confirmed, the server automatically notifies the relevant authorities. The notification is securely transmitted via email or application programming interface.

[0553] On the other hand, when a user interacts with the device, the device uses its camera and microphone to capture the user's facial expressions and voice in real time. By analyzing this data using sentiment analysis APIs such as Google Cloud Vision and IBM Watson, the system identifies the user's emotions. For example, if the system detects impatience or anxiety while a user is looking for a bicycle parking space, the device uses the Google Maps API to guide the user to the nearest available parking space.

[0554] As a concrete example, consider a situation where a user is looking for a bicycle parking space near a train station. The system detects the user's voice and facial expressions, and if it detects "anxiety," the terminal displays "We will guide you to a nearby bicycle parking space" on the screen and provides a function to suggest the nearest possible locations. In this case, the system will respond if the user says, "Tell me about nearby bicycle parking spaces."

[0555] This configuration allows the system to check for parking violations in real time while providing services tailored to the user's emotions, creating a more efficient and user-friendly environment.

[0556] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0557] Step 1:

[0558] The server receives video data in real time from the acquisition device. This data is preprocessed using the OpenCV library. The input is video data from the surveillance camera, and the output is high-quality video data with noise reduction. Specifically, filtering is performed to reduce shadows and noise.

[0559] Step 2:

[0560] The server inputs pre-processed video data into an AI algorithm to detect bicycles. TensorFlow and PyTorch are used for this purpose. The input is pre-processed video data, and the output is identification information indicating the presence of bicycles. Specifically, object detection calculations are performed by the model for each frame.

[0561] Step 3:

[0562] The server compares the location information of identified bicycles with a geographic information database. The PostGIS library is used for this process. The input is location coordinate information obtained through object identification, and the output is the result of determining whether the location is within an illegal parking area. Specifically, a coordinate matching calculation is performed.

[0563] Step 4:

[0564] The server notifies the relevant authorities if it detects illegal parking in a prohibited area. Notifications are sent via email or API. The input is the result of the illegal parking determination, and the output is the logging and transmission of the notification content. Specifically, communication with the mail server takes place.

[0565] Step 5:

[0566] The device captures the user's facial expressions and voice using its camera and microphone and sends them to an emotion engine. This data is analyzed using Google Cloud Vision or IBM Watson. The input is the user's real-time facial expressions and voice, and the output is analyzed emotion information. Specifically, this involves frequency analysis of the voice and feature extraction of facial expressions.

[0567] Step 6:

[0568] The device provides users with appropriate information based on detected emotions. Specifically, it uses the Google Maps API to guide users to nearby bicycle parking spaces. The input is analyzed emotion information, and the output is map information provided to the user. The specific operations include map search and display of results.

[0569] (Application Example 2)

[0570] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] Illegal bicycle parking in urban areas obstructs traffic and hinders the efficient use of public spaces. Furthermore, the inability to find parking spaces can cause stress for users. Given these issues, there is a need for an interactive management system that balances public safety with user convenience.

[0572] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0573] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for identifying an object based on the preprocessed video data and determining whether the identified object is located in a predetermined prohibited area, and means for analyzing the user's psychological state from their face and voice and providing interactive guidance based on the analysis results. This enables not only detection and notification of illegal parking, but also the provision of appropriate parking guidance based on the user's emotions, thereby improving public safety and reducing user stress.

[0574] "Acquisition device" refers to hardware or software used to acquire video data of a monitored object.

[0575] "Preprocessing" refers to the process of performing initial processing to convert received video data into a format that allows for object identification.

[0576] "Object identification" refers to recognizing a specific object from pre-processed video data and extracting its characteristics.

[0577] A "designated prohibited area" refers to a specific geographical area where the presence of an object is not permitted.

[0578] "Analyzing a user's psychological state from their face and voice" is a process that evaluates an individual's emotions or psychological tendencies based on their facial expressions and tone of voice.

[0579] "Providing interactive guidance" means dynamically offering information and services that are tailored to the user's psychological state.

[0580] "Notification" refers to the act of transmitting relevant information to a designated institution based on the judgment or analysis results.

[0581] "Relevant organization" refers to the body or organization responsible for responding when it receives notification.

[0582] The server uses OpenCV and other image recognition libraries to preprocess video data acquired from monitoring devices. This allows it to identify objects such as bicycles and determine if they are in a prohibited area. The object's location information is cross-referenced with a Geographic Information System (GIS) database, and if illegal parking is detected, the system automatically notifies the relevant authorities via email or API.

[0583] The device uses the Emotion Recognition API to analyze the user's facial expressions and voice tone. This analysis is then used to provide information about the optimal parking space if the user is experiencing any psychological distress. By interactively receiving information through the device, the stress of parking is reduced.

[0584] For example, when a user is looking for a bicycle parking space while traveling around a train station, the terminal displays the availability of parking spaces via video data. Furthermore, if it senses impatience from the user's facial expression, it provides prompt guidance tailored to their psychological state. This system aims to improve the efficient management of public spaces in cities and enhance user convenience.

[0585] An example of a prompt message when using a generative AI model is, "Please tell me how to suggest the best parking location when the user is feeling stressed."

[0586] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0587] Step 1:

[0588] The server receives video data transmitted from the acquisition device and performs preprocessing using image recognition libraries such as OpenCV. This process analyzes the video data and converts it into a format suitable for object detection models. The input is raw video data, and the output is preprocessed data suitable for object detection.

[0589] Step 2:

[0590] The server identifies objects (e.g., bicycles) using pre-processed data. By applying an object detection algorithm, it extracts the location information of bicycles within the video. The input is pre-processed data, and the output is data containing the location information of the identified bicycles.

[0591] Step 3:

[0592] The server compares the location information of identified objects with a Geographic Information System (GIS) database and determines whether the objects are located within a restricted area. The input is the location information of the objects, and the output is the result of the area determination.

[0593] Step 4:

[0594] The server notifies the relevant authorities of the presence of an object within a restricted area. This notification is sent automatically via email or an application programming interface (API). The input is the area detection result, and the output is the notification transmission.

[0595] Step 5:

[0596] The device acquires the user's facial and voice data and analyzes their psychological state using the Emotion Recognition API. The input is the user's facial and voice data, and the output is the analyzed emotional information.

[0597] Step 6:

[0598] The device provides interactive guidance to the user based on an analysis of their psychological state. For example, if the user is anxious while searching for a bicycle parking spot, it will present information on the optimal parking location. The input is emotional information, and the output is guidance information for the user.

[0599] Step 7:

[0600] The user adjusts their actions based on guidance from the device. This reduces the stress of choosing a parking spot and moving around. The input is guidance information provided by the device, and the output is the user's actions.

[0601] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0602] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0603] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0604] [Fourth Embodiment]

[0605] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0606] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0607] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0608] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0609] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0610] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0611] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0612] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0613] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0614] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0615] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0616] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0617] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0618] This invention is a system that analyzes video data from an acquisition device, with the aim of rapidly and effectively detecting and reporting illegal parking. The system utilizes AI-powered object recognition technology and a database-based method for determining prohibited areas.

[0619] The server first receives video data from the acquisition device and performs preprocessing on that data. This preprocessing includes noise reduction and image resizing. Next, the server passes the preprocessed data to an AI algorithm for object recognition. This determines where objects such as bicycles are located within the video.

[0620] Subsequently, the server checks the location information of the identified bicycle against a database of prohibited areas to determine if it is illegally parked. Once this determination is made, the server records the information of the illegally parked bicycle and initiates the process of notifying the relevant authorities.

[0621] As a concrete example, imagine a camera installed in a busy area of ​​a city that continuously captures video in real time. The server uses AI to detect bicycles parked at specific locations from this video, and if they are within a designated prohibited area, it automatically records that fact. The server then notifies the police or city hall via email to prompt action against illegal parking.

[0622] The terminal provides an interface that allows users to check information on illegally parked bicycles in real time, helping the general public to park their bicycles responsibly and with awareness.

[0623] Thus, this system is effective in addressing the problem of illegal bicycle parking in urban areas and contributes to improving public safety and comfort.

[0624] The following describes the processing flow.

[0625] Step 1:

[0626] The server receives video data in real time from the acquisition device. This video data is transmitted from cameras designed to continuously cover the area being monitored.

[0627] Step 2:

[0628] The server performs preprocessing on the received video data. Specifically, it removes noise from the video and resizes the resolution to a size suitable for the AI ​​algorithm.

[0629] Step 3:

[0630] The server inputs the pre-processed video into an AI algorithm and begins object recognition. Here, a convolutional neural network (CNN) is used to detect bicycles in the image and determine their location.

[0631] Step 4:

[0632] The server then compares the location information of the identified bicycle with a geographic information database. This allows it to determine whether the identified bicycle is located within a no-parking zone.

[0633] Step 5:

[0634] The server records information in its database if it determines that a bicycle is illegally parked. This record includes the date and time of detection, location information, and a video snapshot.

[0635] Step 6:

[0636] The server automatically generates a notification message based on recorded illegal parking information and sends the notification to the relevant authorities in the appropriate format. This notification is sent via email or a dedicated API.

[0637] Step 7:

[0638] The terminal provides a user interface that allows users to view information on illegally parked bicycles. This enables users to understand the situation of illegal parking in real time and to be more mindful of preventing parking problems.

[0639] (Example 1)

[0640] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0641] Illegal parking of bicycles poses a significant problem for urban traffic, landscape, and crime prevention. Conventional measures require substantial human resources for monitoring and enforcement, making effective control difficult. This invention aims to provide a system that efficiently and quickly detects illegally parked bicycles with limited resources and notifies the relevant authorities.

[0642] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0643] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for performing object identification using a generated AI model with the preprocessed video data, determining the location of the identified object, and determining whether the location matches a predetermined prohibited area database, and means for notifying relevant organizations of information about the identified object based on the determination result via electronic communication means. This makes it possible to automatically detect illegal parking and notify relevant parties quickly and with high accuracy.

[0644] An "acquisition device" is a device used to acquire video data from an object being observed, and includes devices such as cameras and sensors.

[0645] "Preprocessing" refers to the initial processing of received video data, such as noise reduction and resizing.

[0646] A "generative AI model" is a model that uses machine learning algorithms and is a mathematical representation for performing data analysis or object recognition in specific tasks.

[0647] "Object recognition" refers to the technology of detecting and classifying specific objects contained within image data.

[0648] A "restricted area database" is a database that holds information about restrictions in a specific area, including information about geographical boundaries.

[0649] "Electronic communication methods" refer to technologies and protocols used for digital communication via email or networks.

[0650] "Geographic information management technology" refers to technologies for collecting, managing, and analyzing geographic information, and includes GIS (Geographic Information Systems).

[0651] In implementing this invention, the server is primarily responsible for data reception, processing, and notification. The server receives video data in real time from cameras installed throughout the city as acquisition devices. These cameras have appropriate image quality and frame rates, enabling surveillance at various points in the city.

[0652] The server preprocesses the received video data using image processing libraries such as OpenCV. This preprocessing includes noise reduction, image quality improvement, and resolution modification. This processing improves the accuracy of object recognition by the subsequent generative AI model.

[0653] Next, the server applies machine learning algorithms such as YOLO (You Only Look Once) as a "generative AI model." This model quickly and accurately identifies objects such as bicycles present in the video. Then, by extracting the location data of the identified objects, it identifies where illegal parking is occurring.

[0654] The server uses geographic information management technologies such as GIS to compare identified location data with a database of prohibited areas. This database includes prohibited areas pre-defined based on urban planning and transportation policies. This allows the server to immediately determine whether or not illegal parking is present.

[0655] If the assessment confirms illegal parking, the server will use electronic communication methods to send a notification to the relevant authorities via email or a dedicated application. This notification will include the location information of the illegally parked bicycle and related image data, enabling a swift response.

[0656] Furthermore, the terminal provides an interface that allows users to check information on illegally parked bicycles in real time. This terminal enables users to understand the current state of illegal parking and contribute to improving parking etiquette.

[0657] A concrete example would be a process where cameras installed in the city center continuously transmit video footage to a server, which then analyzes the data to immediately identify illegal parking at specific locations and notifies the police and public transport authorities.

[0658] An example of a prompt is, "Explain the procedure for detecting objects from real-time video and comparing them to prohibited areas," which demonstrates an application using a generative AI model.

[0659] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0660] Step 1:

[0661] The server receives video data from the acquisition device. The input is real-time video data from the camera, and by acquiring this data, the server is ready to begin detecting illegally parked bicycles. Specifically, it continuously receives digital video signals via the network.

[0662] Step 2:

[0663] The server performs preprocessing on the received video data. The input is the received video data, and the output is image data with noise removed and the resolution appropriately adjusted. Here, the OpenCV library is used to perform noise reduction and image resizing, thereby preparing the data for subsequent analysis.

[0664] Step 3:

[0665] The server inputs preprocessed image data into a generating AI model to perform object recognition. The input is preprocessed image data, and the output is data identifying the position of bicycles within the image. Specifically, the YOLO model is applied to detect bicycles in each frame and extract their coordinate information.

[0666] Step 4:

[0667] The server uses geographic information management technology (GIS) to compare the location information of identified objects against a prohibited area database. The input is the identified location information, and the output is the result of determining whether the object is within a prohibited area. In this process, the location information is cross-referenced using GIS to determine the possibility of illegal parking.

[0668] Step 5:

[0669] Based on the judgment results, the server notifies the relevant authorities of information about bicycles deemed illegally parked. The input is the result of matching against prohibited areas, and the output is detailed information about the reported illegally parked bicycles. Specifically, it automatically notifies the necessary authorities via email or API and transmits illegal parking information, including image data.

[0670] Step 6:

[0671] The terminal provides users with real-time information on illegally parked bicycles. Input is the reported information received from the server, and output is the illegal parking status displayed on the user interface. This process presents information in a visually easy-to-understand format, allowing users to check it immediately.

[0672] (Application Example 1)

[0673] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0674] Illegal bicycle parking in urban areas is a major problem for city life, as it obstructs traffic, damages the aesthetics of the area, and hinders emergency evacuation routes. However, the current monitoring system suffers from a shortage of personnel and limitations in monitoring area, making it difficult to quickly detect and respond to illegal parking. Furthermore, there is a lack of guidance for citizens to choose appropriate parking locations. Therefore, there is a need to establish a system that enables real-time detection and reporting of illegal parking, as well as guidance to citizens on suitable parking locations.

[0675] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0676] In this invention, the server includes means for preprocessing visual data of a monitored object received from an acquisition device; means for identifying an object based on the preprocessed visual data and determining whether the identified object is located in a predetermined prohibited area; means for notifying relevant organizations of information about the identified object based on the determination result; and means for notifying users of a digital terminal in real time about objects located within the predetermined prohibited area and guiding them to an appropriate parking location. This enables rapid detection and response to illegal parking and guidance of citizens to appropriate parking locations.

[0677] An "acquisition device" is hardware or a system used to acquire visual data of a monitored object.

[0678] "Preprocessing" refers to the process of removing noise and adjusting the size of acquired visual data to convert it into a state suitable for analysis.

[0679] "Object recognition" is a technology that uses algorithms such as AI to detect specific objects present in visual data and to identify what they are.

[0680] A "restricted area" is information indicating a specific area where the presence of objects is not permitted, and this information is cross-referenced with a geographic information database.

[0681] "Related organizations" refer to organizations or entities that receive information about illegal parking and take necessary action.

[0682] "Notification" is the act of transmitting information about an identified object to a designated recipient using electronic means of communication.

[0683] A "digital device" is an electronic device that provides a user interface, and includes smartphones and tablets.

[0684] "Real-time" refers to the ability to process or provide data instantly, without delay.

[0685] "Bicycle parking location guidance" is a service that provides users with information and guidance on appropriate bicycle parking locations.

[0686] This invention comprises a system for detecting and notifying illegal bicycle parking using a digital terminal. The server receives visual data from acquisition devices installed in the monitored area and performs preprocessing. This preprocessing includes noise reduction and image resizing. As a result, the data is appropriately analyzed by an AI algorithm, enabling object identification.

[0687] The server uses TensorFlow or similar methods to perform object recognition and detect objects such as bicycles in the visual data. The identified objects are then compared against prohibited area information stored in a geographic information database, and if a match is found, it is determined to be illegally parked. Information regarding illegal parking is immediately notified to the relevant authorities via electronic communication.

[0688] At the same time, users with digital devices can receive notifications in real time. The devices display on-site information about illegally parked bicycles and also provide guidance to appropriate parking locations. For example, if a citizen tries to park their bicycle near the entrance of a park, the device can issue an alert saying, "This is a no-parking zone," and display information about the nearest parking area along with a map.

[0689] The aim is to enable rapid response to illegal bicycle parking in urban areas and appropriate parking guidance through the user experience. In the system construction based on this invention, object recognition using a generative AI model and instant communication functions play important roles.

[0690] Examples of input prompts for a generative AI model for specific scenarios are as follows:

[0691] "Please explain how to identify bicycles parked in restricted areas using camera footage and generate an alert message when they are detected."

[0692] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0693] Step 1:

[0694] The server receives visual input data from the acquisition device. The received data is acquired as a raw video stream. This data is used as input for processing in the next step.

[0695] Step 2:

[0696] The server performs preprocessing on the received visual data. This preprocessing includes removing noise from the image and resizing it to a size that is easily handled by the AI ​​model. As a result of the preprocessing, appropriately formatted video data is output.

[0697] Step 3:

[0698] The server analyzes the pre-processed video data using an AI model to perform object identification. Specifically, the AI ​​model detects specific objects in the video, such as bicycles. The obtained identification information is generated as output and used in subsequent steps.

[0699] Step 4:

[0700] The server compares the location information of the identified object with a geographic information database. This comparison determines whether the object is located within a no-parking zone. The result of the determination (whether or not illegal parking is present) and the object's location information are output.

[0701] Step 5:

[0702] If illegal parking is detected, the server notifies the relevant authorities of the determination. The notification is made automatically using electronic communication methods, and the content of the report is output.

[0703] Step 6:

[0704] The device notifies the user in real time about objects within the restricted area. The notification is sent to the device, and guidance to a suitable parking location is also provided. This notification allows the user to immediately understand the current situation in the restricted area and take appropriate action.

[0705] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0706] This invention provides a more interactive and comprehensive management system by combining an illegal parking detection and notification system with an emotion engine that recognizes user emotions. This system operates by integrating object recognition capabilities using video data with emotion analysis.

[0707] The server preprocesses the video data received from the acquisition device and uses an AI algorithm to detect bicycles. The server then compares the location of the detected bicycles with a geographic information database to determine if they are illegally parked in a prohibited area. If illegal parking is confirmed, the server notifies the relevant authorities based on the recorded information.

[0708] This system also features an emotion engine that identifies emotions from the user's facial expressions and voice. When a user interacts with the system, the terminal analyzes the user's emotions in real time and records them in a database. For example, if a user looking for a place to park their bicycle is feeling stressed, the emotion engine can identify that emotion and provide information about available parking locations.

[0709] As a concrete example, consider a scenario where a user is using a device to search for a bicycle parking space. In this case, the system analyzes the user's voice and facial expressions using an emotion engine. If it detects anxiety or impatience, the device displays a map and guides the user to nearby available parking spaces. This information can also be shared with relevant organizations for safety management purposes.

[0710] In this way, this system, which incorporates an emotion engine, achieves both a better user experience and improved public safety by integrating object detection and emotion analysis.

[0711] The following describes the processing flow.

[0712] Step 1:

[0713] The server receives video data in real time from the acquisition device. This data captures the situation within the monitoring area.

[0714] Step 2:

[0715] The server performs preprocessing on the received video data. Specifically, it removes noise from the video and adjusts the image to a size that is easy to analyze.

[0716] Step 3:

[0717] The server inputs the pre-processed video data into an AI algorithm to perform object recognition. Here, it detects bicycles and other objects and determines their location.

[0718] Step 4:

[0719] The server then compares the location information of the identified object with a geographic information database to determine whether it is within a restricted area.

[0720] Step 5:

[0721] The server notifies the relevant authorities of details about objects determined to be illegally parked. This notification is made promptly via email or API.

[0722] Step 6:

[0723] The device captures the user's facial expressions and voice and sends them to the emotion engine. This is to analyze the user's emotions in real time while they are interacting with the system.

[0724] Step 7:

[0725] The server analyzes emotional data obtained from the emotion engine and generates feedback tailored to the user's state. For example, if the user is feeling stressed, it will present supportive information.

[0726] Step 8:

[0727] The device displays necessary information to the user based on emotional data. This includes providing specific guidance on areas where bicycles can be parked, as well as messages aimed at addressing the user's emotional needs.

[0728] Through this process, the system can not only manage illegal parking but also respond to users' emotions, aiming to maintain a more interactive and effective urban environment.

[0729] (Example 2)

[0730] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0731] In modern urban environments, illegal bicycle parking is a significant problem that severely impairs public traffic and aesthetics. However, simply monitoring parking is not enough; responses that consider the feelings of users are also required. Therefore, in addition to detecting illegal parking, it is necessary to understand the feelings of users and provide appropriate information accordingly. A comprehensive system is needed to address these complex issues.

[0732] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0733] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for identifying an object based on the preprocessed video data and determining whether the identified object is located in a predetermined prohibited area, and means for analyzing the user's facial expressions and voice to identify their emotions. This enables the detection of illegal parking and the provision of information based on the user's emotions.

[0734] An "acquisition device" is a device that acquires video data from the area being monitored.

[0735] "Video data" refers to data that digitally records the visual information of the subject being monitored.

[0736] "Preprocessing" refers to the initial stage of data processing performed to make video data easier to analyze.

[0737] "Object" refers to an identifiable entity within video data, and in this invention, it primarily refers to a bicycle.

[0738] "Identification" refers to the act of identifying the type and characteristics of an object based on video data.

[0739] A "forbidden area" refers to a designated location where the presence of objects is not permitted.

[0740] "Determination" is the act of determining whether or not an identified object exists in a prohibited area.

[0741] "Relevant body" refers to any organization or body that has the authority to take appropriate action upon receiving notification.

[0742] "Notification" refers to the act of transmitting information, such as judgment results, to other organizations or devices.

[0743] "Facial expressions and voice" refer to the user's visual and auditory expressions of emotion.

[0744] "Emotions" are elements that indicate the user's psychological state and are identified by the system.

[0745] "Information provision" refers to the act of presenting useful data to users.

[0746] This invention integrates a module that analyzes user emotions into a system for detecting and reporting illegally parked bicycles. The system incorporates both object detection and emotion recognition capabilities, aiming to improve the user experience and maintain public order.

[0747] The server receives video data from acquisition devices such as surveillance cameras. This data is preprocessed using the OpenCV library, including noise reduction and resolution adjustment. The preprocessed data is then analyzed by AI algorithms utilizing TensorFlow, PyTorch, etc., to detect bicycles in the video. The location information of the detected objects is cross-referenced with a geographic information database to evaluate whether they fall within a prohibited area. Database tools such as PostGIS are used for this process. If parking in a prohibited area is confirmed, the server automatically notifies the relevant authorities. The notification is securely transmitted via email or application programming interface.

[0748] On the other hand, when a user interacts with the device, the device uses its camera and microphone to capture the user's facial expressions and voice in real time. By analyzing this data using sentiment analysis APIs such as Google Cloud Vision and IBM Watson, the system identifies the user's emotions. For example, if the system detects impatience or anxiety while a user is looking for a bicycle parking space, the device uses the Google Maps API to guide the user to the nearest available parking space.

[0749] As a concrete example, consider a situation where a user is looking for a bicycle parking space near a train station. The system detects the user's voice and facial expressions, and if it detects "anxiety," the terminal displays "We will guide you to a nearby bicycle parking space" on the screen and provides a function to suggest the nearest possible locations. In this case, the system will respond if the user says, "Tell me about nearby bicycle parking spaces."

[0750] This configuration allows the system to check for parking violations in real time while providing services tailored to the user's emotions, creating a more efficient and user-friendly environment.

[0751] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0752] Step 1:

[0753] The server receives video data in real time from the acquisition device. This data is preprocessed using the OpenCV library. The input is video data from the surveillance camera, and the output is high-quality video data with noise reduction. Specifically, filtering is performed to reduce shadows and noise.

[0754] Step 2:

[0755] The server inputs pre-processed video data into an AI algorithm to detect bicycles. TensorFlow and PyTorch are used for this purpose. The input is pre-processed video data, and the output is identification information indicating the presence of bicycles. Specifically, object detection calculations are performed by the model for each frame.

[0756] Step 3:

[0757] The server compares the location information of identified bicycles with a geographic information database. The PostGIS library is used for this process. The input is location coordinate information obtained through object identification, and the output is the result of determining whether the location is within an illegal parking area. Specifically, a coordinate matching calculation is performed.

[0758] Step 4:

[0759] The server notifies the relevant authorities if it detects illegal parking in a prohibited area. Notifications are sent via email or API. The input is the result of the illegal parking determination, and the output is the logging and transmission of the notification content. Specifically, communication with the mail server takes place.

[0760] Step 5:

[0761] The device captures the user's facial expressions and voice using its camera and microphone and sends them to an emotion engine. This data is analyzed using Google Cloud Vision or IBM Watson. The input is the user's real-time facial expressions and voice, and the output is analyzed emotion information. Specifically, this involves frequency analysis of the voice and feature extraction of facial expressions.

[0762] Step 6:

[0763] The device provides users with appropriate information based on detected emotions. Specifically, it uses the Google Maps API to guide users to nearby bicycle parking spaces. The input is analyzed emotion information, and the output is map information provided to the user. The specific operations include map search and display of results.

[0764] (Application Example 2)

[0765] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0766] Illegal bicycle parking in urban areas obstructs traffic and hinders the efficient use of public spaces. Furthermore, the inability to find parking spaces can cause stress for users. Given these issues, there is a need for an interactive management system that balances public safety with user convenience.

[0767] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0768] In this invention, the server includes means for preprocessing video data of a monitored object received from an acquisition device, means for identifying an object based on the preprocessed video data and determining whether the identified object is located in a predetermined prohibited area, and means for analyzing the user's psychological state from their face and voice and providing interactive guidance based on the analysis results. This enables not only detection and notification of illegal parking, but also the provision of appropriate parking guidance based on the user's emotions, thereby improving public safety and reducing user stress.

[0769] "Acquisition device" refers to hardware or software used to acquire video data of a monitored object.

[0770] "Preprocessing" refers to the process of performing initial processing to convert received video data into a format that allows for object identification.

[0771] "Object identification" refers to recognizing a specific object from pre-processed video data and extracting its characteristics.

[0772] A "designated prohibited area" refers to a specific geographical area where the presence of an object is not permitted.

[0773] "Analyzing a user's psychological state from their face and voice" is a process that evaluates an individual's emotions or psychological tendencies based on their facial expressions and tone of voice.

[0774] "Providing interactive guidance" means dynamically offering information and services that are tailored to the user's psychological state.

[0775] "Notification" refers to the act of transmitting relevant information to a designated institution based on the judgment or analysis results.

[0776] "Relevant organization" refers to the body or organization responsible for responding when it receives notification.

[0777] The server uses OpenCV and other image recognition libraries to preprocess video data acquired from monitoring devices. This allows it to identify objects such as bicycles and determine if they are in a prohibited area. The object's location information is cross-referenced with a Geographic Information System (GIS) database, and if illegal parking is detected, the system automatically notifies the relevant authorities via email or API.

[0778] The device uses the Emotion Recognition API to analyze the user's facial expressions and voice tone. This analysis is then used to provide information about the optimal parking space if the user is experiencing any psychological distress. By interactively receiving information through the device, the stress of parking is reduced.

[0779] For example, when a user is looking for a bicycle parking space while traveling around a train station, the terminal displays the availability of parking spaces via video data. Furthermore, if it senses impatience from the user's facial expression, it provides prompt guidance tailored to their psychological state. This system aims to improve the efficient management of public spaces in cities and enhance user convenience.

[0780] An example of a prompt message when using a generative AI model is, "Please tell me how to suggest the best parking location when the user is feeling stressed."

[0781] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0782] Step 1:

[0783] The server receives video data transmitted from the acquisition device and performs preprocessing using image recognition libraries such as OpenCV. This process analyzes the video data and converts it into a format suitable for object detection models. The input is raw video data, and the output is preprocessed data suitable for object detection.

[0784] Step 2:

[0785] The server identifies objects (e.g., bicycles) using pre-processed data. By applying an object detection algorithm, it extracts the location information of bicycles within the video. The input is pre-processed data, and the output is data containing the location information of the identified bicycles.

[0786] Step 3:

[0787] The server compares the location information of identified objects with a Geographic Information System (GIS) database and determines whether the objects are located within a restricted area. The input is the location information of the objects, and the output is the result of the area determination.

[0788] Step 4:

[0789] The server notifies the relevant authorities of the presence of an object within a restricted area. This notification is sent automatically via email or an application programming interface (API). The input is the area detection result, and the output is the notification transmission.

[0790] Step 5:

[0791] The device acquires the user's facial and voice data and analyzes their psychological state using the Emotion Recognition API. The input is the user's facial and voice data, and the output is the analyzed emotional information.

[0792] Step 6:

[0793] The device provides interactive guidance to the user based on an analysis of their psychological state. For example, if the user is anxious while searching for a bicycle parking spot, it will present information on the optimal parking location. The input is emotional information, and the output is guidance information for the user.

[0794] Step 7:

[0795] The user adjusts their actions based on guidance from the device. This reduces the stress of choosing a parking spot and moving around. The input is guidance information provided by the device, and the output is the user's actions.

[0796] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0797] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0798] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0799] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0800] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0801] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0802] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0803] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0804] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0805] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0806] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0807] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0808] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0809] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0810] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0811] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0812] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0813] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0814] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0815] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0816] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0817] The following is further disclosed regarding the embodiments described above.

[0818] (Claim 1)

[0819] A means for preprocessing video data of the monitored object received from the acquisition device,

[0820] A means for identifying an object based on pre-processed video data and determining whether the identified object is located in a predetermined prohibited area,

[0821] A means of notifying relevant organizations of information about the object identified based on the judgment result,

[0822] A system that includes this.

[0823] (Claim 2)

[0824] The system according to claim 1, which compares location information in video data received from an acquisition device with a geographic information database.

[0825] (Claim 3)

[0826] The system according to claim 1, which automatically notifies relevant organizations via email or an application program interface.

[0827] "Example 1"

[0828] (Claim 1)

[0829] A means for preprocessing video data of the monitored object received from the acquisition device,

[0830] A means for performing object identification using a generated AI model with preprocessed video data, determining the location of the identified object, and determining whether that location matches a predetermined prohibited area database,

[0831] A means of notifying relevant organizations of information about the object identified based on the judgment result via electronic communication means,

[0832] A system that includes this.

[0833] (Claim 2)

[0834] The system according to claim 1, wherein location information in video data received from an acquisition device is compared with a prohibited area database using geographic information management technology.

[0835] (Claim 3)

[0836] The system according to claim 1, which automatically notifies relevant organizations using electronic communication means and further records the notification information.

[0837] "Application Example 1"

[0838] (Claim 1)

[0839] Means for preprocessing the visual data of the monitored object received from the acquisition device,

[0840] Means for identifying an object based on pre-processed visual data and determining whether the identified object is located in a predetermined prohibited area,

[0841] A means of notifying relevant organizations of information about the object identified based on the judgment result,

[0842] A means of notifying the user in real time about objects within a designated restricted area via a digital terminal and guiding them to an appropriate parking location.

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, which compares location information in visual data received from an acquisition device with a geographic information database.

[0846] (Claim 3)

[0847] The system according to claim 1, which automatically notifies relevant organizations using an electronic communication method or an applied program interface.

[0848] "Example 2 of combining an emotion engine"

[0849] (Claim 1)

[0850] A means for preprocessing video data of the monitored object received from the acquisition device,

[0851] A means for identifying an object based on pre-processed video data and determining whether the identified object is located in a predetermined prohibited area,

[0852] A means of notifying relevant organizations of information about the object identified based on the judgment result,

[0853] A means of identifying emotions by analyzing the user's facial expressions and voice,

[0854] A means of providing users with appropriate information based on their emotions,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, which compares location information in video data received from an acquisition device with a geographic information database.

[0858] (Claim 3)

[0859] The system according to claim 1, which automatically notifies relevant organizations using electronic communication means.

[0860] "Application example 2 when combining with an emotional engine"

[0861] (Claim 1)

[0862] A means for preprocessing video data of the monitored object received from the acquisition device,

[0863] A means for identifying an object based on pre-processed video data and determining whether the identified object is located in a predetermined prohibited area,

[0864] A means of analyzing the user's psychological state from their face and voice, and providing interactive guidance based on the analysis results,

[0865] Means for notifying relevant organizations of information about objects identified based on judgment results or analysis results, or information about the user's status,

[0866] A system that includes this.

[0867] (Claim 2)

[0868] The system according to claim 1, which compares location information in video data received from an acquisition device with a geographic information database.

[0869] (Claim 3)

[0870] The system according to claim 1, which automatically notifies relevant organizations via email or an application program interface. [Explanation of Symbols]

[0871] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for preprocessing the visual data of the monitored object received from the acquisition device, Means for identifying an object based on pre-processed visual data and determining whether the identified object is located in a predetermined prohibited area, A means of notifying relevant organizations of information about the object identified based on the judgment result, A means of notifying the user in real time about objects within a designated restricted area via a digital terminal and guiding them to an appropriate parking location. A system that includes this.

2. The system according to claim 1, which compares location information in visual data received from an acquisition device with a geographic information database.

3. The system according to claim 1, which automatically notifies relevant organizations using an electronic communication method or an applied program interface.

Images