System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring video data, a means for transmitting the acquired video data to a server, a means for analyzing the video data by the server and generating a determination result, a means for transmitting the generated determination result to a terminal, and a means for notifying a user of the determination result received by the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the shortage of referees in sports competitions has become a serious problem. In particular, in the field of child sports development, the lack of referees has made it difficult to run fair games, leading to increased violence and verbal abuse from parents and coaches, and negatively impacting the healthy development of children. The objective of this invention is to provide a system that solves these problems and ensures the fair and smooth running of sports competitions. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means: a system having means for acquiring video data in real time and transmitting it to a server, means for the server to analyze the received video data and generate a judgment result, and means for transmitting the generated judgment result to a terminal and notifying the user of the judgment result received by the terminal. This system uses AI technology to analyze the situation of a sports game in real time, provide fair judgment, and solve the problem of a shortage of referees.

[0006] "Means for acquiring video data" refers to a device or function for capturing video in real time using a camera, sensor, etc.

[0007] "Means for transmitting captured video data to a server" refers to a process or function for transmitting the captured video data to a server over a network.

[0008] "Means by which the server analyzes video data and generates a judgment result" refers to the process by which the server analyzes the video data it receives using AI technology or machine learning models and determines the judgment content based on the results.

[0009] The "means for transmitting the generated determination result to the terminal" refers to a function or process in which the server transmits data to notify the terminal of the determination result obtained from the analysis.

[0010] "Means for notifying the user of the determination result received by the terminal" refers to the function of the terminal to receive the determination result sent from the server and notify the user of it visually, audibly, or in other ways.

[0011] "Means for capturing video data in real time" refers to a process for continuously obtaining current video using a camera device or other imaging equipment.

[0012] "Means for encoding captured video data into an appropriate format" refers to the process of converting captured video data into a format that can be received and processed by a server or other analysis system.

[0013] "Means for sending encoded data to a server via an HTTP request" refers to a process for sending the converted video data to a server using the HTTP protocol.

[0014] "Means for decoding video data received by the server and analyzing it using an AI model" refers to the process of restoring the encoded data received by the server to its original video format, and inputting that video data into an AI model for analysis.

[0015] "Means of encoding the analysis results in JSON format and sending them to the terminal via an HTTP request" refers to the process of converting the results of the analysis into JSON format and sending that data to the terminal using the HTTP protocol. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention provides a system for automating the role of referees in sports competitions and supporting fair and efficient game management. This system has a series of means for acquiring video data in real time, transmitting the acquired video data to a server, analyzing the received video data and generating a judgment result, transmitting the generated judgment result to a terminal, and notifying the user of the judgment result received by the terminal.

[0038] (System configuration)

[0039] 1. Acquiring video data

[0040] The device (such as a smartphone or AI glasses worn by the user) acquires video data of the game in real time. It uses a camera to continuously capture frames of data and processes them in real-time streaming.

[0041] 2. Transmission of video data

[0042] The device sends the acquired video data to the server, where it performs encoding and sends the encoded data to the server via an HTTP request.

[0043] 3. Analysis of video data

[0044] The server analyzes the received video data and generates a decision. A pre-trained AI model is used for the analysis, and the video data is pre-processed before being input into the AI ​​model. The decision results include whether or not a foul was committed and the scoring situation.

[0045] 4. Sending the results

[0046] The server sends the generated judgment result to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request.

[0047] 5. Notification of the decision

[0048] The device notifies the user of the received judgment result, which may be displayed on a screen or output as audio.

[0049] (Example)

[0050] For example, during a soccer match, a device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action.

[0051] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language during competitions.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] The device initializes the camera device and starts video capture. The device uses the camera to capture successive frames in real time and generate a video stream.

[0055] Step 2:

[0056] The device encodes the received frame data into an appropriate format, such as JPEG, and processes the frame as binary data that can be transmitted.

[0057] Step 3:

[0058] The device sends the encoded video data to the server via an HTTP request. The device checks the status of the transmission to see if it was successful or not, and attempts to retransmit if necessary.

[0059] Step 4:

[0060] The server decodes the video data received via the HTTP request and restores the original video frames from the received binary data.

[0061] Step 5:

[0062] The server preprocesses the decoded video frames to make them suitable for input to a pre-trained AI model, specifically by adjusting and normalizing the frame size.

[0063] Step 6:

[0064] The server then inputs the pre-processed video data into the AI ​​model for analysis, which then makes decisions on fouls and other issues based on the video data.

[0065] Step 7:

[0066] The server generates a decision based on the analysis results of the AI ​​model, such as whether to call a foul or continue playing based on the predicted probability.

[0067] Step 8:

[0068] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. The server confirms that the transmission was successful.

[0069] Step 9:

[0070] The terminal receives the judgment result sent from the server and prepares to notify the user based on the received data.

[0071] Step 10:

[0072] The device then notifies the user of the received decision results. Possible notification methods include displaying the results on a screen or announcing the results by voice, allowing the user to check the results of the match in real time.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Fair and prompt refereeing decisions are important in modern sports competitions. However, problems such as human error and a shortage of referees can have a significant impact on the management of games. Conventional technology has not sufficiently reduced the burden on referees or improved the accuracy of their decisions, so a new system is needed to improve the quality of sports competitions.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes means for decoding received video data and analyzing it using an AI model, means for encoding the analysis results in JSON format and transmitting them to the terminal as judgment results, and means for notifying the user of the judgment results received by the terminal. This enables fair and prompt judgment, reduces the burden on referees, and improves the quality of sports competitions.

[0078] "Video data" is a collection of video frames captured during a sporting event.

[0079] "Preprocessing" refers to processing such as noise removal and frame rate adjustment that is carried out before analyzing video data.

[0080] "Encoding" is the process of compressing and converting video data into a specific format.

[0081] A "server" is a computer system that receives video data, decodes it, and analyzes it using an AI model.

[0082] "Decoding" is the process of restoring encoded data to its original form.

[0083] An "AI model" is a computational model that uses pre-trained algorithms to analyze video data.

[0084] "Analysis" is the process of using an AI model to evaluate the content of video data and derive a judgment result.

[0085] The "decision result" is judgment information obtained as a result of analysis, and indicates, for example, whether or not a foul has been committed.

[0086] The "JSON format" is a lightweight data exchange format for structuring and expressing data such as judgment results.

[0087] "Notification" refers to the action of the terminal notifying the user of the determination result.

[0088] A "terminal" is a device worn or used by a user, and is an electronic device that displays the received judgment results or notifies them by voice.

[0089] The present invention provides a system for automating the role of referees in sports competitions and supporting fair and efficient game management. This system has a series of means for acquiring video data in real time, transmitting the acquired video data to a server, analyzing the received video data and generating a judgment result, transmitting the generated judgment result to a terminal, and notifying the user of the judgment result received by the terminal.

[0090] Specifically, the AI ​​glasses worn by the user or a device such as a smartphone acquires video data during the match in real time. For example, the device's camera continuously captures frame data, which is then processed using real-time streaming technology (such as the RTSP protocol).

[0091] The device preprocesses the captured video data and then performs encoding. Preprocessing involves removing noise from the video data and adjusting the frame rate, for example, converting the video data resolution to 720p. Next, the video data is encoded (for example, using the H.264 codec) and compressed. This encoded data is then sent to the server using an HTTP request.

[0092] The server decodes the received video data and inputs it into a pre-trained AI model. For analysis, it uses machine learning frameworks such as TENSORFLOW (registered trademark) or PyTorch. The server uses the AI ​​model to extract features for each frame and analyzes and detects contact between players and rule violations (fouls).

[0093] The result of the analysis, such as a "foul" judgment, is encoded in JSON format and sent to the terminal via an HTTP request.

[0094] The device will then notify the user of the received decision. Specifically, it may display "Foul" on the device's display or announce "A foul has been detected" through a voice notification. The device may also flash a portion of the screen in red as visual feedback to the user.

[0095] Examples:

[0096] For example, imagine a referee wearing AI glasses during a soccer match. These AI glasses capture video of the match in real time. The captured video data undergoes noise reduction and frame rate adjustment on the device, and is then encoded using the H.264 codec. The encoded data is sent to the server via an HTTP POST request. The server decodes the video data and uses an AI model (e.g., ResNet-50) to analyze the player movements and contact on a frame-by-frame basis. If the analysis determines that contact between players constitutes a rules violation (a foul), the server encodes that information into JSON format and sends it back to the device via an HTTP POST request.

[0097] When the device receives this decision, it will display "Foul" on the screen and announce "A foul has been detected" via voice, allowing the referee (user) to take appropriate action immediately.

[0098] Example prompt sentence:

[0099] "During a soccer match, the AI ​​glasses worn by the user will capture video of the match in real time and send that video data to a server. The server will analyze the video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis result indicates a foul, the decision will be sent to the AI ​​glasses, which will then notify the user (the referee) that a 'foul' has occurred. Please generate a program for this process."

[0100] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0101] Step 1:

[0102] The device acquires video data. Specifically, AI glasses worn by the user or a smartphone camera captures video data of the game in real time. The input is video of the game, and the output is continuous frame data. For example, the device captures video at 30 frames per second. The captured video data is then sent to the next step.

[0103] Step 2:

[0104] The video data captured by the device is preprocessed and then encoded. Preprocessing involves removing noise from the video data and adjusting the frame rate. For example, converting the resolution of the video data to 720p. The input is the captured raw video data, and the output is preprocessed video data. The preprocessed video data is then encoded using the H.264 codec. The input is the preprocessed video data, and the output is encoded compressed video data.

[0105] Step 3:

[0106] The device sends the encoded video data to the server via an HTTP request. The input is the encoded data, and the output is an HTTP request to the server. Specifically, the device sends the following HTTP POST request to the server:

[0107] POST / api / upload HTTP / 1.1

[0108] Host: server.example.com

[0109] Content-Type: application / octet-stream

[0110] Content-Length: [data length]

[0111] [data]

[0112] Step 4:

[0113] The server decodes the received video data and inputs it into the AI ​​model. First, the server decodes the encoded data to restore the original video data. The encoded data is the input, and the decoded video data is the output. Next, the server inputs the decoded video data into a pre-trained AI model (using, for example, TensorFlow or PyTorch) and performs analysis. The decoded video data is the input, and the analysis results are the output.

[0114] Step 5:

[0115] The server generates a verdict based on the analysis results of the AI ​​model and encodes it in JSON format. Specifically, the information obtained as a result of the analysis (such as whether or not there was a foul) is encoded in JSON format. The input is the analysis results of the AI ​​model, and the output is the verdict encoded in JSON format. For example, the JSON data of the verdict looks like this:

[0116] json

[0117] {

[0118] "event": "foul",

[0119] "timestamp": "2023-10-01T15:04:05Z",

[0120] "details": {

[0121] "player": "Player 10",

[0122] "team": "Team A"

[0123] }

[0124] }

[0125] Step 6:

[0126] The server generates a judgment result and sends it to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request. The input is the judgment result in JSON format, and the output is an HTTP request to the terminal. Specifically, the server sends the following HTTP POST request to the terminal:

[0127] POST / api / notification HTTP / 1.1

[0128] Host: terminal.example.com

[0129] Content-Type: application / json

[0130] Content-Length: [data length]

[0131] [JSON data]

[0132] Step 7:

[0133] The device notifies the user of the decision result received. Specifically, it displays "Foul" on the device's display or announces "A foul has been detected" through an audio notification. The input is the decision result received from the server, and the output is a visual or audio notification to the user. The device may also flash a part of the screen red as visual feedback to the user.

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] Conventional security monitoring systems have difficulty detecting suspicious behavior or individuals in real time, and consume a large amount of human resources, especially in environments that require wide-area monitoring. Visual monitoring by on-site security personnel is inefficient and can result in delays in situations that require immediate response. Furthermore, the accuracy of detecting suspicious behavior depends on the viewer's subjective judgment, leading to inconsistent judgments. There is a need to solve these issues and improve the efficiency and accuracy of security monitoring.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server includes a means for analyzing video data and generating a judgment result, a means for using a generative AI model to detect suspicious behaviors and people in real time, and a means for executing the AI ​​model based on a prompt sentence, thereby enabling highly accurate detection of suspicious behaviors and people in real time.

[0139] "Video data" is a collection of visual information generated from a camera or other image capture device.

[0140] A "server" is an information processing device or system that receives and transmits data via a network and performs processes such as analysis and storage.

[0141] A "terminal" is a device that can be directly operated by a user and that transmits and receives data to and from a server. For example, a smartphone or smart glasses fall into this category.

[0142] A "generative AI model" is a model that uses pre-trained artificial intelligence and is capable of performing specified tasks.

[0143] A "prompt" is an instruction or input text that an AI model uses to perform a specific task, guiding the AI ​​to take appropriate action.

[0144] "Real time" means that processing is carried out the moment an event occurs, and indicates a situation in which an immediate response can be made without delay.

[0145] "Suspicious behavior" refers to abnormal behavior that deviates from normal patterns of behavior and may pose a security risk.

[0146] "Person" refers to a human being within a monitored area who is the subject of security assessment and behavior monitoring.

[0147] An "HTTP request" is a protocol used to send and receive data between a web server and a client, and is a means for making a request to a server.

[0148] "Encoding" is the process of converting data into a particular format, and in the case of video data, this means converting it into a digital format that is easier to transmit or store.

[0149] "Decoding" refers to the process of restoring encoded data to its original form, meaning converting the received data into an analyzable format.

[0150] This invention is a system that automates security monitoring and realizes highly accurate detection of suspicious behavior and individuals. This system executes a series of processes: acquisition, transmission, analysis, generation of judgment results, and notification of video data. Each component of the system and its function are described below.

[0151] First, the user's device (e.g., a smartphone or smart glasses) acquires video data in real time from a surveillance camera or the device's own camera. The acquired video data is encoded into an appropriate format (e.g., JPEG) and sent to the server via an HTTP request. This is done using OpenCV and the requests library.

[0152] The server decodes the transmitted video data and analyzes the video using a pre-trained generative AI model (e.g., a model using TensorFlow Keras). This generative AI model is designed to detect suspicious behavior and people with high accuracy. As a result of the analysis, a list of suspicious behavior and people is generated.

[0153] The analysis results are encoded in JSON format and sent to the device again via an HTTP request. The device then notifies the user of the received judgment results. Possible notification methods include displaying the results on a screen or outputting audio. This process allows the user to receive real-time detection results for suspicious behavior and people.

[0154] As a concrete example, consider a surveillance system in a commercial facility. This system analyzes video footage from multiple surveillance cameras in real time to detect theft and suspicious activity. For example, the following prompts could be used:

[0155] "Detect suspicious behavior or people from video data. The subject is surveillance footage from a commercial facility."

[0156] Based on this prompt, the generative AI model performs a specific task and notifies the device of the results.

[0157] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0158] Step 1:

[0159] The device acquires the video data

[0160] The device (smartphone or smart glasses) captures video data in real time using a surveillance camera or built-in camera. The input is the subject of the video effect, and the output is a series of frames of video data. Specifically, it monitors an area specified by the user and captures frames at specified intervals.

[0161] Step 2:

[0162] The device sends the video data to the server

[0163] The captured video data is encoded (e.g., into JPEG format) and sent to the server via an HTTP request. The input is the captured video frame, and the output is the encoded data. Specifically, the video is temporarily stored on the device and then sent to the server via an appropriate network connection.

[0164] Step 3:

[0165] The server receives and decodes the video data

[0166] The server receives the video data sent from the device and performs the decoding process. The input is encoded video data, and the output is video data in an analyzable format. Specifically, the server receives an HTTP request, saves the received data in memory, and uses a decoding library to restore the original video data.

[0167] Step 4:

[0168] The server analyzes the video data based on the generated AI model

[0169] The server inputs the decoded video data into a generative AI model for analysis. The input is the decoded video data, and the output is the detection results of suspicious behavior and people. Specifically, the server uses AI libraries such as TensorFlow and Keras to analyze the input video in real time and execute the generative AI model according to the prompt text.

[0170] Step 5:

[0171] The server sends the analysis results to the terminal in JSON format.

[0172] The server encodes the analysis results in JSON format and sends them to the terminal via an HTTP request. The input is the analysis results, and the output is JSON format data. Specifically, the server encodes the analysis results generated by the server using a JSON library and sends an HTTP request to the address specified by the terminal.

[0173] Step 6:

[0174] The device receives the analysis results and notifies the user.

[0175] The device receives the analysis results in JSON format sent from the server and notifies the user of the contents. The input is the analysis results in JSON format, and the output is a notification to the user (display or audio output). Specifically, the device analyzes the received data and notifies the user of the results of the detection of suspicious behavior or people via the screen display or audio output device.

[0176] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0177] This invention relates to a system that automates the role of referees in sports competitions and supports fair and efficient game management. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to dynamically adjust notification content and user interface.

[0178] (System configuration)

[0179] 1. Acquiring video data

[0180] The device (such as a smartphone or AI glasses worn by the user) acquires video data of the game in real time. It uses a camera to continuously capture frames of data and processes them in real-time streaming.

[0181] 2. Transmission of video data

[0182] The device sends the acquired video data to the server, where it performs encoding and sends the encoded data to the server via an HTTP request.

[0183] 3. Analysis of video data

[0184] The server analyzes the received video data and generates a decision. A pre-trained AI model is used for the analysis, and the video data is pre-processed before being input into the AI ​​model. The decision results include whether or not a foul was committed and the scoring situation.

[0185] 4. Sending the results

[0186] The server sends the generated judgment result to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request.

[0187] 5. Notification of the decision

[0188] The device notifies the user of the received judgment result, which may be displayed on a screen or output as audio.

[0189] 6. Emotion Engine

[0190] The emotion engine has a means for recognizing the user's emotions in real time. The device captures the user's facial expressions and voice and sends the data to the emotion engine. The emotion engine analyzes the user's emotional state and adjusts the notification content based on the analysis results.

[0191] (Example)

[0192] For example, during a soccer match, a device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action.

[0193] In addition, the emotion engine recognizes the user's emotions, and if the user expresses, for example, impatience or excitement, the notification content is changed to a more specific and appropriate method, helping the user make quick and appropriate decisions. This system enables fair and prompt judgment, reduces the burden on referees, and improves the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language in competitions.

[0194] The processing flow will be explained below.

[0195] Step 1:

[0196] The device initializes the camera device and starts video capture. The device uses the camera to capture successive frames in real time and generate a video stream.

[0197] Step 2:

[0198] The device encodes the received frame data into an appropriate format, such as JPEG, and processes the frame as binary data that can be transmitted.

[0199] Step 3:

[0200] The device sends the encoded video data to the server via an HTTP request. The device checks the status of the transmission to see if it was successful or not, and attempts to retransmit if necessary.

[0201] Step 4:

[0202] The server decodes the video data received via the HTTP request and restores the original video frames from the received binary data.

[0203] Step 5:

[0204] The server preprocesses the decoded video frames to make them suitable for input to a pre-trained AI model, specifically by adjusting and normalizing the frame size.

[0205] Step 6:

[0206] The server then inputs the pre-processed video data into the AI ​​model for analysis, which then makes decisions on fouls and other issues based on the video data.

[0207] Step 7:

[0208] The server generates a decision based on the analysis results of the AI ​​model, such as whether to call a foul or continue playing based on the predicted probability.

[0209] Step 8:

[0210] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. The server confirms that the transmission was successful.

[0211] Step 9:

[0212] The terminal receives the judgment result sent from the server and prepares to notify the user based on the received data.

[0213] Step 10:

[0214] The device captures the user's facial expressions and voice data, which are then sent to the emotion engine in real time. The emotion engine analyzes the data to recognize the user's emotional state (e.g., joy, anger, sadness, etc.).

[0215] Step 11:

[0216] The emotion engine determines the user's emotions based on the analysis results, and classifies the results as emotional states such as "normal," "excited," and "tense."

[0217] Step 12:

[0218] The device combines the results of the judgment and the emotion engine to determine the final notification content. For example, if the user is in an "excited" state in the market, the notification content will be adjusted to allow the user to process the situation more calmly.

[0219] Step 13:

[0220] The device will display the final judgement results on its screen and, if necessary, provide an audio notification, allowing users to use this information to ensure fair game management and make appropriate decisions immediately.

[0221] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language during competitions.

[0222] Example 2

[0223] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0224] Judgments made by referees can be heavily subjective and lack fairness. Furthermore, referees face a significant psychological and physical burden, particularly in sports where quick and accurate decisions must be made in real time. Their skills and physical condition can have a significant impact. Furthermore, the issue of a shortage of referees and problems such as violence and verbal abuse during competitions cannot be ignored. To solve these issues, a system is needed that uses video data to automatically make objective and prompt decisions.

[0225] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0226] In this invention, the server includes means for analyzing video data and generating a judgment result, means for transmitting the generated judgment result to the terminal, and means for analyzing the user's emotions and dynamically adjusting the notification content based on the analysis result, which enables fair and prompt game management, reduces the burden on referees, and improves the quality of sports competitions.

[0227] "Video Data" means video information of a game or competition captured using a camera or other video capture device.

[0228] A "server" is a computer system in a network that receives video data, analyzes it, and generates and transmits a judgment result.

[0229] A "terminal" is a device worn or used by a user (e.g., AI glasses or a smartphone), which receives the judgment results from the server and notifies the user.

[0230] "User" refers to the referees and other competition organizers who use this system and receive judgement results and notifications via their terminals.

[0231] The "emotion engine" is a mechanism that analyzes the user's facial expressions and voice data to recognize the user's emotional state and dynamically adjusts the content of notifications.

[0232] "Real-time" refers to processing or communication that requires minimal delay and immediacy.

[0233] "Encoding" is the process of converting raw data into an efficient format, and in this context refers specifically to the compression of video data using a video codec.

[0234] An "HTTP request" is a communication request based on a protocol for sending and receiving data between a client and a server.

[0235] The "JSON format" is a lightweight data exchange format for structuring and representing data.

[0236] A "generative AI model" is a pre-trained artificial intelligence algorithm that is used to analyze video data and generate judgment results.

[0237] This invention relates to a system that automates the role of referees in sports competitions and supports fair and efficient game management. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to dynamically adjust notification content and user interface.

[0238] The main components of the system are the terminal, the server, and the emotion engine. Specifically, the following hardware and software are used:

[0239] Device: Refers to the AI ​​glasses or smartphone worn by the user, which includes a camera and video processing application for capturing game video data in real time. The device encodes the video data using a video codec such as H.264 and sends the data to the server via an HTTP request.

[0240] Server: A computer system that receives and analyzes video data. The server uses a generative AI model built using machine learning frameworks such as TensorFlow or PyTorch to generate a judgment result from the video data. The judgment result is encoded in JSON format and sent to the device via an HTTP request.

[0241] Emotion engine: A system that analyzes the user's facial expressions and voice to recognize their emotional state. It uses emotion analysis APIs from IBM Watson (registered trademark) and Microsoft (registered trademark) Azure (registered trademark), and dynamically adjusts notification content based on the results.

[0242] (Example)

[0243] For example, during a soccer match, the device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server then analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device then notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action. The emotion engine also recognizes the user's emotions, and if the user expresses feelings such as impatience or excitement, the notification content is changed to a more specific and appropriate method, helping the user to make quick and appropriate decisions.

[0244] (Example of a prompt)

[0245] "Please explain the system in which a device worn by the user (for example, AI glasses or a smartphone) acquires video data of the game in real time and determines whether or not a foul has been committed. Also, please specify the components of the system that recognizes the user's emotions and adjusts the notification content in accordance with the guidelines."

[0246] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language in competitions.

[0247] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0248] Step 1:

[0249] The device acquires video data of the game in real time. The camera continuously captures frame data and temporarily stores it in memory. Specifically, the device's camera captures video frames every 30 seconds and stores the frame data in memory. The input of this step is the actual video of the game, and the output is the captured frame data.

[0250] Step 2:

[0251] The device encodes the captured video data using the H.264 video codec. The encoding process is performed efficiently using the device's CPU and GPU. Specifically, the captured video frames are compressed and converted into a format suitable for transmission. The input to this step is the captured frame data, and the output is the encoded video data.

[0252] Step 3:

[0253] The device sends the encoded video data to the server via an HTTP request. The HTTP request is generated every 3 seconds and contains the encoded data as a payload. Specifically, the data is temporarily stored in a send buffer and transmitted over a network connection (e.g., Wi-Fi or LTE). The input of this step is the encoded video data, and the output is an HTTP request.

[0254] Step 4:

[0255] The server decodes and preprocesses the received video data. Preprocessing includes noise reduction and resolution adjustment. Specifically, it applies filters to each frame to ensure data quality. The input of this step is the encoded video data sent in the HTTP request, and the output is the preprocessed video data.

[0256] Step 5:

[0257] The server inputs the preprocessed video data into a generative AI model for analysis. This analysis uses pre-trained AI algorithms to determine player movements for each frame. For example, it determines whether contact is a rule violation. The input for this step is the preprocessed video data, and the output is the decision result.

[0258] Step 6:

[0259] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. Specifically, it converts the judgment result into a data structure and generates and sends an HTTP response. The input to this step is the judgment result, and the output is data encoded in JSON format.

[0260] Step 7:

[0261] The device analyzes the received decision result and notifies the user. The notification method is to show the result on the screen or notify the user audibly through the speaker. Specifically, the decision result, such as "foul," is communicated to the user visually and audibly. The input to this step is the decision result in JSON format, and the output is a notification to the user.

[0262] Step 8:

[0263] The device captures the user's facial expressions and voice and sends them to the emotion engine. The emotion engine analyzes facial expressions and voice to recognize the user's emotional state. Specifically, it determines emotions such as impatience or excitement from facial expressions. The input to this step is the user's facial expression and voice data, and the output is the emotion analysis results.

[0264] Step 9:

[0265] The server adjusts the notification content based on the emotion analysis results. For example, if the user is excited, it generates an additional message such as "Please stay calm." The input of this step is the emotion analysis results, and the output is the adjusted notification content.

[0266] Step 10:

[0267] The device re-notifies the user of the adjusted notification content. Specifically, it conveys specific and appropriate instructions to the user by displaying them on the display or outputting them as audio from the speaker. The input of this step is the adjusted notification content, and the output is the notification to the user.

[0268] The above processing steps enable fair and efficient game management.

[0269] (Application example 2)

[0270] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0271] In today's manufacturing workplaces, product quality control and monitoring of workers' emotional states are highly valued, but performing these tasks efficiently can be difficult. Quality inspections and evaluations of workers' emotional states, in particular, rely on human judgment and involve subjective factors, which can lead to a lack of fairness and accuracy. Furthermore, properly understanding workers' fatigue and stress and encouraging them to take necessary breaks is important from the perspective of improving productivity and preventing accidents. Therefore, there is a need for fair and accurate quality control and the establishment of a system that can monitor and evaluate workers' emotional states in real time.

[0272] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data to the server, means for the server to analyze the video data and generate a determination result, means for transmitting the generated determination result to the terminal, means for notifying the user of the determination result received by the terminal, means for dynamically adjusting the notification content using an emotion engine that recognizes the user's emotions, means for performing quality control in a factory environment using an artificial intelligence model, and means for analyzing the video data based on prompt sentences to determine the emotions of workers. This enables fair and accurate quality control, and also makes it possible to grasp the emotional state of workers in real time and provide appropriate notifications or interventions as needed.

[0273] "Video data" refers to information about moving images acquired using devices such as cameras and sensors.

[0274] A "server" is a computer system that transmits, receives, and processes data over a network.

[0275] A "terminal" is a device that is operated by a user and has the function of displaying or notifying received information.

[0276] An "emotion engine" is an artificial intelligence technology that analyzes data such as a user's facial expressions and voice to recognize their emotional state in real time.

[0277] An "artificial intelligence model" is an algorithm that learns from large amounts of data and performs specific tasks.

[0278] "Quality control" is a set of activities and processes for maintaining the quality of products and services to a certain standard.

[0279] A "generative AI model" is an artificial intelligence model that uses techniques such as generative adversarial networks (GANs) and natural language processing (NLP) to generate new data and text.

[0280] A "prompt sentence" is an instruction sentence input to a generative AI model that determines the direction of the output generated by the model.

[0281] The system for implementing this invention mainly comprises a set of means for acquiring video data, analyzing it, and generating and notifying results. It also utilizes an emotion engine and artificial intelligence model to monitor quality control and the emotional state of workers in a factory environment.

[0282] System Configuration

[0283] 1. Acquiring video data

[0284] The device uses a camera to capture video data in real time. This video data can be used for, for example, product quality inspections or monitoring the work environment. The camera can be a general webcam or an industrial camera.

[0285] 2. Data transmission

[0286] The device encodes the acquired video data into an appropriate format and sends it to the server via an HTTP request, using a library such as the requests library.

[0287] 3. Analysis of video data

[0288] The server decodes the received video data and analyzes it using a pre-trained generative AI model. Specifically, it detects defects for quality control and determines the emotional state of workers. This analysis uses machine learning libraries such as PyTorch and TensorFlow.

[0289] 4. Generating and sending analysis results

[0290] The server encodes the analysis results in JSON format and sends them to the terminal via an HTTP request. The analysis results include information such as "poor quality" and "worker stress level."

[0291] 5. Result Notification and Emotion Engine

[0292] The device then notifies the user of the received judgment results by displaying a message on the screen or issuing a voice alert. The device also utilizes an emotion engine that recognizes the user's emotions in real time and dynamically adjusts the content of the notification based on the analysis results.

[0293] Specific examples

[0294] For example, consider a worker wearing smart glasses inspecting products. The system detects defective products in real time and displays a message to the worker, such as "A defective product has been detected." If the emotion engine detects stress in the worker, it will send an appropriate notification, such as "You are feeling stressed. We recommend that you take a short break." This enables fair and accurate quality control while improving worker safety and work efficiency.

[0295] Specific hardware and software used

[0296] Camera: General webcam, industrial camera

[0297] Devices: Smartphones, smart glasses, etc.

[0298] Server: High-performance computing server (e.g., AWS (registered trademark) EC2, Google (registered trademark) Cloud Platform)

[0299] Libraries: requests (data transmission), cv2 (video capture), PyTorch or TensorFlow (AI model analysis)

[0300] Prompt Sentence Examples

[0301] Analyze input images to determine if they contain defects, and analyze the user's emotional state to recognize stress or fatigue and provide appropriate notifications accordingly.

[0302] This invention significantly improves work efficiency and quality control at manufacturing sites and reduces the burden on workers.

[0303] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0304] Step 1:

[0305] The device acquires video data in real time using a camera. Specifically, it continuously captures frame data from the camera device. The acquired video data is input in frame format (e.g., JPEG).

[0306] Step 2:

[0307] The video data acquired by the device is encoded into an appropriate format (e.g., JPEG). The encoded data is then processed into a format that is easy to send, and becomes output data to be sent to the server in the form of an HTTP request. This process is performed using cv2.imencode or similar.

[0308] Step 3:

[0309] The device sends the encoded video data to the server using an HTTP request. Specifically, it sends the encoded frame data to the server using a method such as requests.post. The input is the encoded video data, and the output is the result of the data transmission to the server.

[0310] Step 4:

[0311] The server decodes the video data it receives. Specifically, it uses functions such as cv2.imdecode on the server side to restore the encoded data back to frame data. The input is the encoded video data, and the output is the decoded frame data.

[0312] Step 5:

[0313] The server analyzes the received video data using a generative AI model. Specifically, using libraries such as PyTorch and TensorFlow, the frame data is input into the AI ​​model to generate a judgment result. The input here is the decoded video data, and the output is the analysis result (e.g., whether there is a quality problem or the emotional state).

[0314] Step 6:

[0315] The server encodes the analysis results into JSON format and sends them to the terminal via an HTTP request. Specifically, it encodes the analysis results into JSON format using json.dumps or similar and sends them via an HTTP request. The input is the analysis results and the output is JSON format data.

[0316] Step 7:

[0317] The device notifies the user of the result of the judgment received. Specifically, it displays a message on the screen or uses an audio alert. The input is the analysis result in JSON format, and the output is a notification to the user (e.g., text message, audio message).

[0318] Step 8:

[0319] The device uses an emotion engine that recognizes the user's emotions in real time. Specifically, the user's facial expressions and voice data are input into the emotion engine, which analyzes their emotional state. The input is the user's facial expressions and voice data, and the output is the emotion analysis results.

[0320] Step 9:

[0321] The device dynamically adjusts the notification content based on the emotion analysis results. For example, if the user is under stress, the notification message will be changed to "We recommend you take a break." The input is the emotion analysis results, and the output is the adjusted notification content.

[0322] Through these steps, the system achieves fair and accurate quality control and real-time monitoring and evaluation of workers' emotional states.

[0323] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0324] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0325] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0326] [Second embodiment]

[0327] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0328] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0329] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0330] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0331] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0332] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0333] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0334] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0335] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0336] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0337] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0338] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0339] The present invention provides a system for automating the role of referees in sports competitions and supporting fair and efficient game management. This system has a series of means for acquiring video data in real time, transmitting the acquired video data to a server, analyzing the received video data and generating a judgment result, transmitting the generated judgment result to a terminal, and notifying the user of the judgment result received by the terminal.

[0340] (System configuration)

[0341] 1. Acquiring video data

[0342] The device (such as a smartphone or AI glasses worn by the user) acquires video data of the game in real time. It uses a camera to continuously capture frames of data and processes them in real-time streaming.

[0343] 2. Transmission of video data

[0344] The device sends the acquired video data to the server, where it performs encoding and sends the encoded data to the server via an HTTP request.

[0345] 3. Analysis of video data

[0346] The server analyzes the received video data and generates a decision. A pre-trained AI model is used for the analysis, and the video data is pre-processed before being input into the AI ​​model. The decision results include whether or not a foul was committed and the scoring situation.

[0347] 4. Sending the results

[0348] The server sends the generated judgment result to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request.

[0349] 5. Notification of the decision

[0350] The device notifies the user of the received judgment result, which may be displayed on a screen or output as audio.

[0351] (Example)

[0352] For example, during a soccer match, a device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action.

[0353] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language during competitions.

[0354] The processing flow will be explained below.

[0355] Step 1:

[0356] The device initializes the camera device and starts video capture. The device uses the camera to capture successive frames in real time and generate a video stream.

[0357] Step 2:

[0358] The device encodes the received frame data into an appropriate format, such as JPEG, and processes the frame as binary data that can be transmitted.

[0359] Step 3:

[0360] The device sends the encoded video data to the server via an HTTP request. The device checks the status of the transmission to see if it was successful or not, and attempts to retransmit if necessary.

[0361] Step 4:

[0362] The server decodes the video data received via the HTTP request and restores the original video frames from the received binary data.

[0363] Step 5:

[0364] The server preprocesses the decoded video frames to make them suitable for input to a pre-trained AI model, specifically by adjusting and normalizing the frame size.

[0365] Step 6:

[0366] The server then inputs the pre-processed video data into the AI ​​model for analysis, which then makes decisions on fouls and other issues based on the video data.

[0367] Step 7:

[0368] The server generates a decision based on the analysis results of the AI ​​model, such as whether to call a foul or continue playing based on the predicted probability.

[0369] Step 8:

[0370] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. The server confirms that the transmission was successful.

[0371] Step 9:

[0372] The terminal receives the judgment result sent from the server and prepares to notify the user based on the received data.

[0373] Step 10:

[0374] The device then notifies the user of the received decision results. Possible notification methods include displaying the results on a screen or announcing the results by voice, allowing the user to check the results of the match in real time.

[0375] Example 1

[0376] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0377] Fair and prompt refereeing decisions are important in modern sports competitions. However, problems such as human error and a shortage of referees can have a significant impact on the management of games. Conventional technology has not sufficiently reduced the burden on referees or improved the accuracy of their decisions, so a new system is needed to improve the quality of sports competitions.

[0378] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0379] In this invention, the server includes means for decoding received video data and analyzing it using an AI model, means for encoding the analysis results in JSON format and transmitting them to the terminal as judgment results, and means for notifying the user of the judgment results received by the terminal. This enables fair and prompt judgment, reduces the burden on referees, and improves the quality of sports competitions.

[0380] "Video data" is a collection of video frames captured during a sporting event.

[0381] "Preprocessing" refers to processing such as noise removal and frame rate adjustment that is carried out before analyzing video data.

[0382] "Encoding" is the process of compressing and converting video data into a specific format.

[0383] A "server" is a computer system that receives video data, decodes it, and analyzes it using an AI model.

[0384] "Decoding" is the process of restoring encoded data to its original form.

[0385] An "AI model" is a computational model that uses pre-trained algorithms to analyze video data.

[0386] "Analysis" is the process of using an AI model to evaluate the content of video data and derive a judgment result.

[0387] The "decision result" is judgment information obtained as a result of analysis, and indicates, for example, whether or not a foul has been committed.

[0388] The "JSON format" is a lightweight data exchange format for structuring and expressing data such as judgment results.

[0389] "Notification" refers to the action of the terminal notifying the user of the determination result.

[0390] A "terminal" is a device worn or used by a user, and is an electronic device that displays the received judgment results or notifies them by voice.

[0391] The present invention provides a system for automating the role of referees in sports competitions and supporting fair and efficient game management. This system has a series of means for acquiring video data in real time, transmitting the acquired video data to a server, analyzing the received video data and generating a judgment result, transmitting the generated judgment result to a terminal, and notifying the user of the judgment result received by the terminal.

[0392] Specifically, the AI ​​glasses worn by the user or a device such as a smartphone acquires video data during the match in real time. For example, the device's camera continuously captures frame data, which is then processed using real-time streaming technology (such as the RTSP protocol).

[0393] The device preprocesses the captured video data and then performs encoding. Preprocessing involves removing noise from the video data and adjusting the frame rate, for example, converting the video data resolution to 720p. Next, the video data is encoded (for example, using the H.264 codec) and compressed. This encoded data is then sent to the server using an HTTP request.

[0394] The server decodes the received video data and inputs it into a pre-trained AI model. For analysis, it uses machine learning frameworks such as TensorFlow and PyTorch. The server uses the AI ​​model to extract features for each frame and analyze and detect contact between players and rule violations (fouls).

[0395] The result of the analysis, such as a "foul" judgment, is encoded in JSON format and sent to the terminal via an HTTP request.

[0396] The device will then notify the user of the received decision. Specifically, it may display "Foul" on the device's display or announce "A foul has been detected" through a voice notification. The device may also flash a portion of the screen in red as visual feedback to the user.

[0397] Examples:

[0398] For example, imagine a referee wearing AI glasses during a soccer match. These AI glasses capture video of the match in real time. The captured video data undergoes noise reduction and frame rate adjustment on the device, and is then encoded using the H.264 codec. The encoded data is sent to the server via an HTTP POST request. The server decodes the video data and uses an AI model (e.g., ResNet-50) to analyze the player movements and contact on a frame-by-frame basis. If the analysis determines that contact between players constitutes a rules violation (a foul), the server encodes that information into JSON format and sends it back to the device via an HTTP POST request.

[0399] When the device receives this decision, it will display "Foul" on the screen and announce "A foul has been detected" via voice, allowing the referee (user) to take appropriate action immediately.

[0400] Example prompt sentence:

[0401] "During a soccer match, the AI ​​glasses worn by the user will capture video of the match in real time and send that video data to a server. The server will analyze the video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis result indicates a foul, the decision will be sent to the AI ​​glasses, which will then notify the user (the referee) that a 'foul' has occurred. Please generate a program for this process."

[0402] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0403] Step 1:

[0404] The device acquires video data. Specifically, AI glasses worn by the user or a smartphone camera captures video data of the game in real time. The input is video of the game, and the output is continuous frame data. For example, the device captures video at 30 frames per second. The captured video data is then sent to the next step.

[0405] Step 2:

[0406] The video data captured by the device is preprocessed and then encoded. Preprocessing involves removing noise from the video data and adjusting the frame rate. For example, converting the resolution of the video data to 720p. The input is the captured raw video data, and the output is preprocessed video data. The preprocessed video data is then encoded using the H.264 codec. The input is the preprocessed video data, and the output is encoded compressed video data.

[0407] Step 3:

[0408] The device sends the encoded video data to the server via an HTTP request. The input is the encoded data, and the output is an HTTP request to the server. Specifically, the device sends the following HTTP POST request to the server:

[0409] POST / api / upload HTTP / 1.1

[0410] Host: server.example.com

[0411] Content-Type: application / octet-stream

[0412] Content-Length: [data length]

[0413] [data]

[0414] Step 4:

[0415] The server decodes the received video data and inputs it into the AI ​​model. First, the server decodes the encoded data to restore the original video data. The encoded data is the input, and the decoded video data is the output. Next, the server inputs the decoded video data into a pre-trained AI model (using, for example, TensorFlow or PyTorch) and performs analysis. The decoded video data is the input, and the analysis results are the output.

[0416] Step 5:

[0417] The server generates a verdict based on the analysis results of the AI ​​model and encodes it in JSON format. Specifically, the information obtained as a result of the analysis (such as whether or not there was a foul) is encoded in JSON format. The input is the analysis results of the AI ​​model, and the output is the verdict encoded in JSON format. For example, the JSON data of the verdict looks like this:

[0418] json

[0419] {

[0420] "event": "foul",

[0421] "timestamp": "2023-10-01T15:04:05Z",

[0422] "details": {

[0423] "player": "Player 10",

[0424] "team": "Team A"

[0425] }

[0426] }

[0427] Step 6:

[0428] The server generates a judgment result and sends it to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request. The input is the judgment result in JSON format, and the output is an HTTP request to the terminal. Specifically, the server sends the following HTTP POST request to the terminal:

[0429] POST / api / notification HTTP / 1.1

[0430] Host: terminal.example.com

[0431] Content-Type: application / json

[0432] Content-Length: [data length]

[0433] [JSON data]

[0434] Step 7:

[0435] The device notifies the user of the decision result received. Specifically, it displays "Foul" on the device's display or announces "A foul has been detected" through an audio notification. The input is the decision result received from the server, and the output is a visual or audio notification to the user. The device may also flash a part of the screen red as visual feedback to the user.

[0436] (Application example 1)

[0437] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0438] Conventional security monitoring systems have difficulty detecting suspicious behavior or individuals in real time, and consume a large amount of human resources, especially in environments that require wide-area monitoring. Visual monitoring by on-site security personnel is inefficient and can result in delays in situations that require immediate response. Furthermore, the accuracy of detecting suspicious behavior depends on the viewer's subjective judgment, leading to inconsistent judgments. There is a need to solve these issues and improve the efficiency and accuracy of security monitoring.

[0439] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0440] In this invention, the server includes a means for analyzing video data and generating a judgment result, a means for using a generative AI model to detect suspicious behaviors and people in real time, and a means for executing the AI ​​model based on a prompt sentence, thereby enabling highly accurate detection of suspicious behaviors and people in real time.

[0441] "Video data" is a collection of visual information generated from a camera or other image capture device.

[0442] A "server" is an information processing device or system that receives and transmits data via a network and performs processes such as analysis and storage.

[0443] A "terminal" is a device that can be directly operated by a user and that transmits and receives data to and from a server. For example, a smartphone or smart glasses fall into this category.

[0444] A "generative AI model" is a model that uses pre-trained artificial intelligence and is capable of performing specified tasks.

[0445] A "prompt" is an instruction or input text that an AI model uses to perform a specific task, guiding the AI ​​to take appropriate action.

[0446] "Real time" means that processing is carried out the moment an event occurs, and indicates a situation in which an immediate response can be made without delay.

[0447] "Suspicious behavior" refers to abnormal behavior that deviates from normal patterns of behavior and may pose a security risk.

[0448] "Person" refers to a human being within a monitored area who is the subject of security assessment and behavior monitoring.

[0449] An "HTTP request" is a protocol used to send and receive data between a web server and a client, and is a means for making a request to a server.

[0450] "Encoding" is the process of converting data into a particular format, and in the case of video data, this means converting it into a digital format that is easier to transmit or store.

[0451] "Decoding" refers to the process of restoring encoded data to its original form, meaning converting the received data into an analyzable format.

[0452] This invention is a system that automates security monitoring and realizes highly accurate detection of suspicious behavior and individuals. This system executes a series of processes: acquisition, transmission, analysis, generation of judgment results, and notification of video data. Each component of the system and its function are described below.

[0453] First, the user's device (e.g., a smartphone or smart glasses) acquires video data in real time from a surveillance camera or the device's own camera. The acquired video data is encoded into an appropriate format (e.g., JPEG) and sent to the server via an HTTP request. This is done using OpenCV and the requests library.

[0454] The server decodes the transmitted video data and analyzes the video using a pre-trained generative AI model (e.g., a model using TensorFlow Keras). This generative AI model is designed to detect suspicious behavior and people with high accuracy. As a result of the analysis, a list of suspicious behavior and people is generated.

[0455] The analysis results are encoded in JSON format and sent to the device again via an HTTP request. The device then notifies the user of the received judgment results. Possible notification methods include displaying the results on a screen or outputting audio. This process allows the user to receive real-time detection results for suspicious behavior and people.

[0456] As a concrete example, consider a surveillance system in a commercial facility. This system analyzes video footage from multiple surveillance cameras in real time to detect theft and suspicious activity. For example, the following prompts could be used:

[0457] "Detect suspicious behavior or people from video data. The subject is surveillance footage from a commercial facility."

[0458] Based on this prompt, the generative AI model performs a specific task and notifies the device of the results.

[0459] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0460] Step 1:

[0461] The device acquires the video data

[0462] The device (smartphone or smart glasses) captures video data in real time using a surveillance camera or built-in camera. The input is the subject of the video effect, and the output is a series of frames of video data. Specifically, it monitors an area specified by the user and captures frames at specified intervals.

[0463] Step 2:

[0464] The device sends the video data to the server

[0465] The captured video data is encoded (e.g., into JPEG format) and sent to the server via an HTTP request. The input is the captured video frame, and the output is the encoded data. Specifically, the video is temporarily stored on the device and then sent to the server via an appropriate network connection.

[0466] Step 3:

[0467] The server receives and decodes the video data

[0468] The server receives the video data sent from the device and performs the decoding process. The input is encoded video data, and the output is video data in an analyzable format. Specifically, the server receives an HTTP request, saves the received data in memory, and uses a decoding library to restore the original video data.

[0469] Step 4:

[0470] The server analyzes the video data based on the generated AI model

[0471] The server inputs the decoded video data into a generative AI model for analysis. The input is the decoded video data, and the output is the detection results of suspicious behavior and people. Specifically, the server uses AI libraries such as TensorFlow and Keras to analyze the input video in real time and execute the generative AI model according to the prompt text.

[0472] Step 5:

[0473] The server sends the analysis results to the terminal in JSON format.

[0474] The server encodes the analysis results in JSON format and sends them to the terminal via an HTTP request. The input is the analysis results, and the output is JSON format data. Specifically, the server encodes the analysis results generated by the server using a JSON library and sends an HTTP request to the address specified by the terminal.

[0475] Step 6:

[0476] The device receives the analysis results and notifies the user.

[0477] The device receives the analysis results in JSON format sent from the server and notifies the user of the contents. The input is the analysis results in JSON format, and the output is a notification to the user (display or audio output). Specifically, the device analyzes the received data and notifies the user of the results of the detection of suspicious behavior or people via the screen display or audio output device.

[0478] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0479] This invention relates to a system that automates the role of referees in sports competitions and supports fair and efficient game management. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to dynamically adjust notification content and user interface.

[0480] (System configuration)

[0481] 1. Acquiring video data

[0482] The device (such as a smartphone or AI glasses worn by the user) acquires video data of the game in real time. It uses a camera to continuously capture frames of data and processes them in real-time streaming.

[0483] 2. Transmission of video data

[0484] The device sends the acquired video data to the server, where it performs encoding and sends the encoded data to the server via an HTTP request.

[0485] 3. Analysis of video data

[0486] The server analyzes the received video data and generates a decision. A pre-trained AI model is used for the analysis, and the video data is pre-processed before being input into the AI ​​model. The decision results include whether or not a foul was committed and the scoring situation.

[0487] 4. Sending the results

[0488] The server sends the generated judgment result to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request.

[0489] 5. Notification of the decision

[0490] The device notifies the user of the received judgment result, which may be displayed on a screen or output as audio.

[0491] 6. Emotion Engine

[0492] The emotion engine has a means for recognizing the user's emotions in real time. The device captures the user's facial expressions and voice and sends the data to the emotion engine. The emotion engine analyzes the user's emotional state and adjusts the notification content based on the analysis results.

[0493] (Example)

[0494] For example, during a soccer match, a device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action.

[0495] In addition, the emotion engine recognizes the user's emotions, and if the user expresses, for example, impatience or excitement, the notification content is changed to a more specific and appropriate method, helping the user make quick and appropriate decisions. This system enables fair and prompt judgment, reduces the burden on referees, and improves the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language in competitions.

[0496] The processing flow will be explained below.

[0497] Step 1:

[0498] The device initializes the camera device and starts video capture. The device uses the camera to capture successive frames in real time and generate a video stream.

[0499] Step 2:

[0500] The device encodes the received frame data into an appropriate format, such as JPEG, and processes the frame as binary data that can be transmitted.

[0501] Step 3:

[0502] The device sends the encoded video data to the server via an HTTP request. The device checks the status of the transmission to see if it was successful or not, and attempts to retransmit if necessary.

[0503] Step 4:

[0504] The server decodes the video data received via the HTTP request and restores the original video frames from the received binary data.

[0505] Step 5:

[0506] The server preprocesses the decoded video frames to make them suitable for input to a pre-trained AI model, specifically by adjusting and normalizing the frame size.

[0507] Step 6:

[0508] The server then inputs the pre-processed video data into the AI ​​model for analysis, which then makes decisions on fouls and other issues based on the video data.

[0509] Step 7:

[0510] The server generates a decision based on the analysis results of the AI ​​model, such as whether to call a foul or continue playing based on the predicted probability.

[0511] Step 8:

[0512] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. The server confirms that the transmission was successful.

[0513] Step 9:

[0514] The terminal receives the judgment result sent from the server and prepares to notify the user based on the received data.

[0515] Step 10:

[0516] The device captures the user's facial expressions and voice data, which are then sent to the emotion engine in real time. The emotion engine analyzes the data to recognize the user's emotional state (e.g., joy, anger, sadness, etc.).

[0517] Step 11:

[0518] The emotion engine determines the user's emotions based on the analysis results, and classifies the results as emotional states such as "normal," "excited," and "tense."

[0519] Step 12:

[0520] The device combines the results of the judgment and the emotion engine to determine the final notification content. For example, if the user is in an "excited" state in the market, the notification content will be adjusted to allow the user to process the situation more calmly.

[0521] Step 13:

[0522] The device will display the final judgement results on its screen and, if necessary, provide an audio notification, allowing users to use this information to ensure fair game management and make appropriate decisions immediately.

[0523] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language during competitions.

[0524] Example 2

[0525] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0526] Judgments made by referees can be heavily subjective and lack fairness. Furthermore, referees face a significant psychological and physical burden, particularly in sports where quick and accurate decisions must be made in real time. Their skills and physical condition can have a significant impact. Furthermore, the issue of a shortage of referees and problems such as violence and verbal abuse during competitions cannot be ignored. To solve these issues, a system is needed that uses video data to automatically make objective and prompt decisions.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0528] In this invention, the server includes means for analyzing video data and generating a judgment result, means for transmitting the generated judgment result to the terminal, and means for analyzing the user's emotions and dynamically adjusting the notification content based on the analysis result, which enables fair and prompt game management, reduces the burden on referees, and improves the quality of sports competitions.

[0529] "Video Data" means video information of a game or competition captured using a camera or other video capture device.

[0530] A "server" is a computer system in a network that receives video data, analyzes it, and generates and transmits a judgment result.

[0531] A "terminal" is a device worn or used by a user (e.g., AI glasses or a smartphone), which receives the judgment results from the server and notifies the user.

[0532] "User" refers to the referees and other competition organizers who use this system and receive judgement results and notifications via their terminals.

[0533] The "emotion engine" is a mechanism that analyzes the user's facial expressions and voice data to recognize the user's emotional state and dynamically adjusts the content of notifications.

[0534] "Real-time" refers to processing or communication that requires minimal delay and immediacy.

[0535] "Encoding" is the process of converting raw data into an efficient format, and in this context refers specifically to the compression of video data using a video codec.

[0536] An "HTTP request" is a communication request based on a protocol for sending and receiving data between a client and a server.

[0537] The "JSON format" is a lightweight data exchange format for structuring and representing data.

[0538] A "generative AI model" is a pre-trained artificial intelligence algorithm that is used to analyze video data and generate judgment results.

[0539] This invention relates to a system that automates the role of referees in sports competitions and supports fair and efficient game management. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to dynamically adjust notification content and user interface.

[0540] The main components of the system are the terminal, the server, and the emotion engine. Specifically, the following hardware and software are used:

[0541] Device: Refers to the AI ​​glasses or smartphone worn by the user, which includes a camera and video processing application for capturing game video data in real time. The device encodes the video data using a video codec such as H.264 and sends the data to the server via an HTTP request.

[0542] Server: A computer system that receives and analyzes video data. The server uses a generative AI model built using machine learning frameworks such as TensorFlow or PyTorch to generate a judgment result from the video data. The judgment result is encoded in JSON format and sent to the device via an HTTP request.

[0543] Emotion engine: A system that analyzes the user's facial expressions and voice to recognize their emotional state. It uses emotion analysis APIs from IBM Watson and Microsoft Azure, and dynamically adjusts notification content based on the results.

[0544] (Example)

[0545] For example, during a soccer match, the device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server then analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device then notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action. The emotion engine also recognizes the user's emotions, and if the user expresses feelings such as impatience or excitement, the notification content is changed to a more specific and appropriate method, helping the user to make quick and appropriate decisions.

[0546] (Example of a prompt)

[0547] "Please explain the system in which a device worn by the user (for example, AI glasses or a smartphone) acquires video data of the game in real time and determines whether or not a foul has been committed. Also, please specify the components of the system that recognizes the user's emotions and adjusts the notification content in accordance with the guidelines."

[0548] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language in competitions.

[0549] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0550] Step 1:

[0551] The device acquires video data of the game in real time. The camera continuously captures frame data and temporarily stores it in memory. Specifically, the device's camera captures video frames every 30 seconds and stores the frame data in memory. The input of this step is the actual video of the game, and the output is the captured frame data.

[0552] Step 2:

[0553] The device encodes the captured video data using the H.264 video codec. The encoding process is performed efficiently using the device's CPU and GPU. Specifically, the captured video frames are compressed and converted into a format suitable for transmission. The input to this step is the captured frame data, and the output is the encoded video data.

[0554] Step 3:

[0555] The device sends the encoded video data to the server via an HTTP request. The HTTP request is generated every 3 seconds and contains the encoded data as a payload. Specifically, the data is temporarily stored in a send buffer and transmitted over a network connection (e.g., Wi-Fi or LTE). The input of this step is the encoded video data, and the output is an HTTP request.

[0556] Step 4:

[0557] The server decodes and preprocesses the received video data. Preprocessing includes noise reduction and resolution adjustment. Specifically, it applies filters to each frame to ensure data quality. The input of this step is the encoded video data sent in the HTTP request, and the output is the preprocessed video data.

[0558] Step 5:

[0559] The server inputs the preprocessed video data into a generative AI model for analysis. This analysis uses pre-trained AI algorithms to determine player movements for each frame. For example, it determines whether contact is a rule violation. The input for this step is the preprocessed video data, and the output is the decision result.

[0560] Step 6:

[0561] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. Specifically, it converts the judgment result into a data structure and generates and sends an HTTP response. The input to this step is the judgment result, and the output is data encoded in JSON format.

[0562] Step 7:

[0563] The device analyzes the received decision result and notifies the user. The notification method is to show the result on the screen or notify the user audibly through the speaker. Specifically, the decision result, such as "foul," is communicated to the user visually and audibly. The input to this step is the decision result in JSON format, and the output is a notification to the user.

[0564] Step 8:

[0565] The device captures the user's facial expressions and voice and sends them to the emotion engine. The emotion engine analyzes facial expressions and voice to recognize the user's emotional state. Specifically, it determines emotions such as impatience or excitement from facial expressions. The input to this step is the user's facial expression and voice data, and the output is the emotion analysis results.

[0566] Step 9:

[0567] The server adjusts the notification content based on the emotion analysis results. For example, if the user is excited, it generates an additional message such as "Please stay calm." The input of this step is the emotion analysis results, and the output is the adjusted notification content.

[0568] Step 10:

[0569] The device re-notifies the user of the adjusted notification content. Specifically, it conveys specific and appropriate instructions to the user by displaying them on the display or outputting them as audio from the speaker. The input of this step is the adjusted notification content, and the output is the notification to the user.

[0570] The above processing steps enable fair and efficient game management.

[0571] (Application example 2)

[0572] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0573] In today's manufacturing workplaces, product quality control and monitoring of workers' emotional states are highly valued, but performing these tasks efficiently can be difficult. Quality inspections and evaluations of workers' emotional states, in particular, rely on human judgment and involve subjective factors, which can lead to a lack of fairness and accuracy. Furthermore, properly understanding workers' fatigue and stress and encouraging them to take necessary breaks is important from the perspective of improving productivity and preventing accidents. Therefore, there is a need for fair and accurate quality control and the establishment of a system that can monitor and evaluate workers' emotional states in real time.

[0574] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data to the server, means for the server to analyze the video data and generate a determination result, means for transmitting the generated determination result to the terminal, means for notifying the user of the determination result received by the terminal, means for dynamically adjusting the notification content using an emotion engine that recognizes the user's emotions, means for performing quality control in a factory environment using an artificial intelligence model, and means for analyzing the video data based on prompt sentences to determine the emotions of workers. This enables fair and accurate quality control, and also makes it possible to grasp the emotional state of workers in real time and provide appropriate notifications or interventions as needed.

[0575] "Video data" refers to information about moving images acquired using devices such as cameras and sensors.

[0576] A "server" is a computer system that transmits, receives, and processes data over a network.

[0577] A "terminal" is a device that is operated by a user and has the function of displaying or notifying received information.

[0578] An "emotion engine" is an artificial intelligence technology that analyzes data such as a user's facial expressions and voice to recognize their emotional state in real time.

[0579] An "artificial intelligence model" is an algorithm that learns from large amounts of data and performs specific tasks.

[0580] "Quality control" is a set of activities and processes for maintaining the quality of products and services to a certain standard.

[0581] A "generative AI model" is an artificial intelligence model that uses techniques such as generative adversarial networks (GANs) and natural language processing (NLP) to generate new data and text.

[0582] A "prompt sentence" is an instruction sentence input to a generative AI model that determines the direction of the output generated by the model.

[0583] The system for implementing this invention mainly comprises a set of means for acquiring video data, analyzing it, and generating and notifying results. It also utilizes an emotion engine and artificial intelligence model to monitor quality control and the emotional state of workers in a factory environment.

[0584] System Configuration

[0585] 1. Acquiring video data

[0586] The device uses a camera to capture video data in real time. This video data can be used for, for example, product quality inspections or monitoring the work environment. The camera can be a general webcam or an industrial camera.

[0587] 2. Data transmission

[0588] The device encodes the acquired video data into an appropriate format and sends it to the server via an HTTP request, using a library such as the requests library.

[0589] 3. Analysis of video data

[0590] The server decodes the received video data and analyzes it using a pre-trained generative AI model. Specifically, it detects defects for quality control and determines the emotional state of workers. This analysis uses machine learning libraries such as PyTorch and TensorFlow.

[0591] 4. Generating and sending analysis results

[0592] The server encodes the analysis results in JSON format and sends them to the terminal via an HTTP request. The analysis results include information such as "poor quality" and "worker stress level."

[0593] 5. Result Notification and Emotion Engine

[0594] The device then notifies the user of the received judgment results by displaying a message on the screen or issuing a voice alert. The device also utilizes an emotion engine that recognizes the user's emotions in real time and dynamically adjusts the content of the notification based on the analysis results.

[0595] Specific examples

[0596] For example, consider a worker wearing smart glasses inspecting products. The system detects defective products in real time and displays a message to the worker, such as "A defective product has been detected." If the emotion engine detects stress in the worker, it will send an appropriate notification, such as "You are feeling stressed. We recommend that you take a short break." This enables fair and accurate quality control while improving worker safety and work efficiency.

[0597] Specific hardware and software used

[0598] Camera: General webcam, industrial camera

[0599] Devices: Smartphones, smart glasses, etc.

[0600] Server: High-performance computing server (e.g. AWS EC2, Google Cloud Platform)

[0601] Libraries: requests (data transmission), cv2 (video capture), PyTorch or TensorFlow (AI model analysis)

[0602] Prompt Sentence Examples

[0603] Analyze input images to determine if they contain defects, and analyze the user's emotional state to recognize stress or fatigue and provide appropriate notifications accordingly.

[0604] This invention significantly improves work efficiency and quality control at manufacturing sites and reduces the burden on workers.

[0605] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0606] Step 1:

[0607] The device acquires video data in real time using a camera. Specifically, it continuously captures frame data from the camera device. The acquired video data is input in frame format (e.g., JPEG).

[0608] Step 2:

[0609] The video data acquired by the device is encoded into an appropriate format (e.g., JPEG). The encoded data is then processed into a format that is easy to send, and becomes output data to be sent to the server in the form of an HTTP request. This process is performed using cv2.imencode or similar.

[0610] Step 3:

[0611] The device sends the encoded video data to the server using an HTTP request. Specifically, it sends the encoded frame data to the server using a method such as requests.post. The input is the encoded video data, and the output is the result of the data transmission to the server.

[0612] Step 4:

[0613] The server decodes the video data it receives. Specifically, it uses functions such as cv2.imdecode on the server side to restore the encoded data back to frame data. The input is the encoded video data, and the output is the decoded frame data.

[0614] Step 5:

[0615] The server analyzes the received video data using a generative AI model. Specifically, using libraries such as PyTorch and TensorFlow, the frame data is input into the AI ​​model to generate a judgment result. The input here is the decoded video data, and the output is the analysis result (e.g., whether there is a quality problem or the emotional state).

[0616] Step 6:

[0617] The server encodes the analysis results into JSON format and sends them to the terminal via an HTTP request. Specifically, it encodes the analysis results into JSON format using json.dumps or similar and sends them via an HTTP request. The input is the analysis results and the output is JSON format data.

[0618] Step 7:

[0619] The device notifies the user of the result of the judgment received. Specifically, it displays a message on the screen or uses an audio alert. The input is the analysis result in JSON format, and the output is a notification to the user (e.g., text message, audio message).

[0620] Step 8:

[0621] The device uses an emotion engine that recognizes the user's emotions in real time. Specifically, the user's facial expressions and voice data are input into the emotion engine, which analyzes their emotional state. The input is the user's facial expressions and voice data, and the output is the emotion analysis results.

[0622] Step 9:

[0623] The device dynamically adjusts the notification content based on the emotion analysis results. For example, if the user is under stress, the notification message will be changed to "We recommend you take a break." The input is the emotion analysis results, and the output is the adjusted notification content.

[0624] Through these steps, the system achieves fair and accurate quality control and real-time monitoring and evaluation of workers' emotional states.

[0625] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0626] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0627] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0628] [Third embodiment]

[0629] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0630] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0631] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0632] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0633] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0635] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0636] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0637] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0638] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0639] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0640] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0641] The present invention provides a system for automating the role of referees in sports competitions and supporting fair and efficient game management. This system has a series of means for acquiring video data in real time, transmitting the acquired video data to a server, analyzing the received video data and generating a judgment result, transmitting the generated judgment result to a terminal, and notifying the user of the judgment result received by the terminal.

[0642] (System configuration)

[0643] 1. Acquiring video data

[0644] The device (such as a smartphone or AI glasses worn by the user) acquires video data of the game in real time. It uses a camera to continuously capture frames of data and processes them in real-time streaming.

[0645] 2. Transmission of video data

[0646] The device sends the acquired video data to the server, where it performs encoding and sends the encoded data to the server via an HTTP request.

[0647] 3. Analysis of video data

[0648] The server analyzes the received video data and generates a decision. A pre-trained AI model is used for the analysis, and the video data is pre-processed before being input into the AI ​​model. The decision results include whether or not a foul was committed and the scoring situation.

[0649] 4. Sending the results

[0650] The server sends the generated judgment result to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request.

[0651] 5. Notification of the decision

[0652] The device notifies the user of the received judgment result, which may be displayed on a screen or output as audio.

[0653] (Example)

[0654] For example, during a soccer match, a device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action.

[0655] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language during competitions.

[0656] The processing flow will be explained below.

[0657] Step 1:

[0658] The device initializes the camera device and starts video capture. The device uses the camera to capture successive frames in real time and generate a video stream.

[0659] Step 2:

[0660] The device encodes the received frame data into an appropriate format, such as JPEG, and processes the frame as binary data that can be transmitted.

[0661] Step 3:

[0662] The device sends the encoded video data to the server via an HTTP request. The device checks the status of the transmission to see if it was successful or not, and attempts to retransmit if necessary.

[0663] Step 4:

[0664] The server decodes the video data received via the HTTP request and restores the original video frames from the received binary data.

[0665] Step 5:

[0666] The server preprocesses the decoded video frames to make them suitable for input to a pre-trained AI model, specifically by adjusting and normalizing the frame size.

[0667] Step 6:

[0668] The server then inputs the pre-processed video data into the AI ​​model for analysis, which then makes decisions on fouls and other issues based on the video data.

[0669] Step 7:

[0670] The server generates a decision based on the analysis results of the AI ​​model, such as whether to call a foul or continue playing based on the predicted probability.

[0671] Step 8:

[0672] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. The server confirms that the transmission was successful.

[0673] Step 9:

[0674] The terminal receives the judgment result sent from the server and prepares to notify the user based on the received data.

[0675] Step 10:

[0676] The device then notifies the user of the received decision results. Possible notification methods include displaying the results on a screen or announcing the results by voice, allowing the user to check the results of the match in real time.

[0677] Example 1

[0678] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0679] Fair and prompt refereeing decisions are important in modern sports competitions. However, problems such as human error and a shortage of referees can have a significant impact on the management of games. Conventional technology has not sufficiently reduced the burden on referees or improved the accuracy of their decisions, so a new system is needed to improve the quality of sports competitions.

[0680] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0681] In this invention, the server includes means for decoding received video data and analyzing it using an AI model, means for encoding the analysis results in JSON format and transmitting them to the terminal as judgment results, and means for notifying the user of the judgment results received by the terminal. This enables fair and prompt judgment, reduces the burden on referees, and improves the quality of sports competitions.

[0682] "Video data" is a collection of video frames captured during a sporting event.

[0683] "Preprocessing" refers to processing such as noise removal and frame rate adjustment that is carried out before analyzing video data.

[0684] "Encoding" is the process of compressing and converting video data into a specific format.

[0685] A "server" is a computer system that receives video data, decodes it, and analyzes it using an AI model.

[0686] "Decoding" is the process of restoring encoded data to its original form.

[0687] An "AI model" is a computational model that uses pre-trained algorithms to analyze video data.

[0688] "Analysis" is the process of using an AI model to evaluate the content of video data and derive a judgment result.

[0689] The "decision result" is judgment information obtained as a result of analysis, and indicates, for example, whether or not a foul has been committed.

[0690] The "JSON format" is a lightweight data exchange format for structuring and expressing data such as judgment results.

[0691] "Notification" refers to the action of the terminal notifying the user of the determination result.

[0692] A "terminal" is a device worn or used by a user, and is an electronic device that displays the received judgment results or notifies them by voice.

[0693] The present invention provides a system for automating the role of referees in sports competitions and supporting fair and efficient game management. This system has a series of means for acquiring video data in real time, transmitting the acquired video data to a server, analyzing the received video data and generating a judgment result, transmitting the generated judgment result to a terminal, and notifying the user of the judgment result received by the terminal.

[0694] Specifically, the AI ​​glasses worn by the user or a device such as a smartphone acquires video data during the match in real time. For example, the device's camera continuously captures frame data, which is then processed using real-time streaming technology (such as the RTSP protocol).

[0695] The device preprocesses the captured video data and then performs encoding. Preprocessing involves removing noise from the video data and adjusting the frame rate, for example, converting the video data resolution to 720p. Next, the video data is encoded (for example, using the H.264 codec) and compressed. This encoded data is then sent to the server using an HTTP request.

[0696] The server decodes the received video data and inputs it into a pre-trained AI model. For analysis, it uses machine learning frameworks such as TensorFlow and PyTorch. The server uses the AI ​​model to extract features for each frame and analyze and detect contact between players and rule violations (fouls).

[0697] The result of the analysis, such as a "foul" judgment, is encoded in JSON format and sent to the terminal via an HTTP request.

[0698] The device will then notify the user of the received decision. Specifically, it may display "Foul" on the device's display or announce "A foul has been detected" through a voice notification. The device may also flash a portion of the screen in red as visual feedback to the user.

[0699] Examples:

[0700] For example, imagine a referee wearing AI glasses during a soccer match. These AI glasses capture video of the match in real time. The captured video data undergoes noise reduction and frame rate adjustment on the device, and is then encoded using the H.264 codec. The encoded data is sent to the server via an HTTP POST request. The server decodes the video data and uses an AI model (e.g., ResNet-50) to analyze the player movements and contact on a frame-by-frame basis. If the analysis determines that contact between players constitutes a rules violation (a foul), the server encodes that information into JSON format and sends it back to the device via an HTTP POST request.

[0701] When the device receives this decision, it will display "Foul" on the screen and announce "A foul has been detected" via voice, allowing the referee (user) to take appropriate action immediately.

[0702] Example prompt sentence:

[0703] "During a soccer match, the AI ​​glasses worn by the user will capture video of the match in real time and send that video data to a server. The server will analyze the video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis result indicates a foul, the decision will be sent to the AI ​​glasses, which will then notify the user (the referee) that a 'foul' has occurred. Please generate a program for this process."

[0704] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0705] Step 1:

[0706] The device acquires video data. Specifically, AI glasses worn by the user or a smartphone camera captures video data of the game in real time. The input is video of the game, and the output is continuous frame data. For example, the device captures video at 30 frames per second. The captured video data is then sent to the next step.

[0707] Step 2:

[0708] The video data captured by the device is preprocessed and then encoded. Preprocessing involves removing noise from the video data and adjusting the frame rate. For example, converting the resolution of the video data to 720p. The input is the captured raw video data, and the output is preprocessed video data. The preprocessed video data is then encoded using the H.264 codec. The input is the preprocessed video data, and the output is encoded compressed video data.

[0709] Step 3:

[0710] The device sends the encoded video data to the server via an HTTP request. The input is the encoded data, and the output is an HTTP request to the server. Specifically, the device sends the following HTTP POST request to the server:

[0711] POST / api / upload HTTP / 1.1

[0712] Host: server.example.com

[0713] Content-Type: application / octet-stream

[0714] Content-Length: [data length]

[0715] [data]

[0716] Step 4:

[0717] The server decodes the received video data and inputs it into the AI ​​model. First, the server decodes the encoded data to restore the original video data. The encoded data is the input, and the decoded video data is the output. Next, the server inputs the decoded video data into a pre-trained AI model (using, for example, TensorFlow or PyTorch) and performs analysis. The decoded video data is the input, and the analysis results are the output.

[0718] Step 5:

[0719] The server generates a verdict based on the analysis results of the AI ​​model and encodes it in JSON format. Specifically, the information obtained as a result of the analysis (such as whether or not there was a foul) is encoded in JSON format. The input is the analysis results of the AI ​​model, and the output is the verdict encoded in JSON format. For example, the JSON data of the verdict looks like this:

[0720] json

[0721] {

[0722] "event": "foul",

[0723] "timestamp": "2023-10-01T15:04:05Z",

[0724] "details": {

[0725] "player": "Player 10",

[0726] "team": "Team A"

[0727] }

[0728] }

[0729] Step 6:

[0730] The server generates a judgment result and sends it to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request. The input is the judgment result in JSON format, and the output is an HTTP request to the terminal. Specifically, the server sends the following HTTP POST request to the terminal:

[0731] POST / api / notification HTTP / 1.1

[0732] Host: terminal.example.com

[0733] Content-Type: application / json

[0734] Content-Length: [data length]

[0735] [JSON data]

[0736] Step 7:

[0737] The device notifies the user of the decision result received. Specifically, it displays "Foul" on the device's display or announces "A foul has been detected" through an audio notification. The input is the decision result received from the server, and the output is a visual or audio notification to the user. The device may also flash a part of the screen red as visual feedback to the user.

[0738] (Application example 1)

[0739] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0740] Conventional security monitoring systems have difficulty detecting suspicious behavior or individuals in real time, and consume a large amount of human resources, especially in environments that require wide-area monitoring. Visual monitoring by on-site security personnel is inefficient and can result in delays in situations that require immediate response. Furthermore, the accuracy of detecting suspicious behavior depends on the viewer's subjective judgment, leading to inconsistent judgments. There is a need to solve these issues and improve the efficiency and accuracy of security monitoring.

[0741] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0742] In this invention, the server includes a means for analyzing video data and generating a judgment result, a means for using a generative AI model to detect suspicious behaviors and people in real time, and a means for executing the AI ​​model based on a prompt sentence, thereby enabling highly accurate detection of suspicious behaviors and people in real time.

[0743] "Video data" is a collection of visual information generated from a camera or other image capture device.

[0744] A "server" is an information processing device or system that receives and transmits data via a network and performs processes such as analysis and storage.

[0745] A "terminal" is a device that can be directly operated by a user and that transmits and receives data to and from a server. For example, a smartphone or smart glasses fall into this category.

[0746] A "generative AI model" is a model that uses pre-trained artificial intelligence and is capable of performing specified tasks.

[0747] A "prompt" is an instruction or input text that an AI model uses to perform a specific task, guiding the AI ​​to take appropriate action.

[0748] "Real time" means that processing is carried out the moment an event occurs, and indicates a situation in which an immediate response can be made without delay.

[0749] "Suspicious behavior" refers to abnormal behavior that deviates from normal patterns of behavior and may pose a security risk.

[0750] "Person" refers to a human being within a monitored area who is the subject of security assessment and behavior monitoring.

[0751] An "HTTP request" is a protocol used to send and receive data between a web server and a client, and is a means for making a request to a server.

[0752] "Encoding" is the process of converting data into a particular format, and in the case of video data, this means converting it into a digital format that is easier to transmit or store.

[0753] "Decoding" refers to the process of restoring encoded data to its original form, meaning converting the received data into an analyzable format.

[0754] This invention is a system that automates security monitoring and realizes highly accurate detection of suspicious behavior and individuals. This system executes a series of processes: acquisition, transmission, analysis, generation of judgment results, and notification of video data. Each component of the system and its function are described below.

[0755] First, the user's device (e.g., a smartphone or smart glasses) acquires video data in real time from a surveillance camera or the device's own camera. The acquired video data is encoded into an appropriate format (e.g., JPEG) and sent to the server via an HTTP request. This is done using OpenCV and the requests library.

[0756] The server decodes the transmitted video data and analyzes the video using a pre-trained generative AI model (e.g., a model using TensorFlow Keras). This generative AI model is designed to detect suspicious behavior and people with high accuracy. As a result of the analysis, a list of suspicious behavior and people is generated.

[0757] The analysis results are encoded in JSON format and sent to the device again via an HTTP request. The device then notifies the user of the received judgment results. Possible notification methods include displaying the results on a screen or outputting audio. This process allows the user to receive real-time detection results for suspicious behavior and people.

[0758] As a concrete example, consider a surveillance system in a commercial facility. This system analyzes video footage from multiple surveillance cameras in real time to detect theft and suspicious activity. For example, the following prompts could be used:

[0759] "Detect suspicious behavior or people from video data. The subject is surveillance footage from a commercial facility."

[0760] Based on this prompt, the generative AI model performs a specific task and notifies the device of the results.

[0761] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0762] Step 1:

[0763] The device acquires the video data

[0764] The device (smartphone or smart glasses) captures video data in real time using a surveillance camera or built-in camera. The input is the subject of the video effect, and the output is a series of frames of video data. Specifically, it monitors an area specified by the user and captures frames at specified intervals.

[0765] Step 2:

[0766] The device sends the video data to the server

[0767] The captured video data is encoded (e.g., into JPEG format) and sent to the server via an HTTP request. The input is the captured video frame, and the output is the encoded data. Specifically, the video is temporarily stored on the device and then sent to the server via an appropriate network connection.

[0768] Step 3:

[0769] The server receives and decodes the video data

[0770] The server receives the video data sent from the device and performs the decoding process. The input is encoded video data, and the output is video data in an analyzable format. Specifically, the server receives an HTTP request, saves the received data in memory, and uses a decoding library to restore the original video data.

[0771] Step 4:

[0772] The server analyzes the video data based on the generated AI model

[0773] The server inputs the decoded video data into a generative AI model for analysis. The input is the decoded video data, and the output is the detection results of suspicious behavior and people. Specifically, the server uses AI libraries such as TensorFlow and Keras to analyze the input video in real time and execute the generative AI model according to the prompt text.

[0774] Step 5:

[0775] The server sends the analysis results to the terminal in JSON format.

[0776] The server encodes the analysis results in JSON format and sends them to the terminal via an HTTP request. The input is the analysis results, and the output is JSON format data. Specifically, the server encodes the analysis results generated by the server using a JSON library and sends an HTTP request to the address specified by the terminal.

[0777] Step 6:

[0778] The device receives the analysis results and notifies the user.

[0779] The device receives the analysis results in JSON format sent from the server and notifies the user of the contents. The input is the analysis results in JSON format, and the output is a notification to the user (display or audio output). Specifically, the device analyzes the received data and notifies the user of the results of the detection of suspicious behavior or people via the screen display or audio output device.

[0780] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0781] This invention relates to a system that automates the role of referees in sports competitions and supports fair and efficient game management. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to dynamically adjust notification content and user interface.

[0782] (System configuration)

[0783] 1. Acquiring video data

[0784] The device (such as a smartphone or AI glasses worn by the user) acquires video data of the game in real time. It uses a camera to continuously capture frames of data and processes them in real-time streaming.

[0785] 2. Transmission of video data

[0786] The device sends the acquired video data to the server, where it performs encoding and sends the encoded data to the server via an HTTP request.

[0787] 3. Analysis of video data

[0788] The server analyzes the received video data and generates a decision. A pre-trained AI model is used for the analysis, and the video data is pre-processed before being input into the AI ​​model. The decision results include whether or not a foul was committed and the scoring situation.

[0789] 4. Sending the results

[0790] The server sends the generated judgment result to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request.

[0791] 5. Notification of the decision

[0792] The device notifies the user of the received judgment result, which may be displayed on a screen or output as audio.

[0793] 6. Emotion Engine

[0794] The emotion engine has a means for recognizing the user's emotions in real time. The device captures the user's facial expressions and voice and sends the data to the emotion engine. The emotion engine analyzes the user's emotional state and adjusts the notification content based on the analysis results.

[0795] (Example)

[0796] For example, during a soccer match, a device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action.

[0797] In addition, the emotion engine recognizes the user's emotions, and if the user expresses, for example, impatience or excitement, the notification content is changed to a more specific and appropriate method, helping the user make quick and appropriate decisions. This system enables fair and prompt judgment, reduces the burden on referees, and improves the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language in competitions.

[0798] The processing flow will be explained below.

[0799] Step 1:

[0800] The device initializes the camera device and starts video capture. The device uses the camera to capture successive frames in real time and generate a video stream.

[0801] Step 2:

[0802] The device encodes the received frame data into an appropriate format, such as JPEG, and processes the frame as binary data that can be transmitted.

[0803] Step 3:

[0804] The device sends the encoded video data to the server via an HTTP request. The device checks the status of the transmission to see if it was successful or not, and attempts to retransmit if necessary.

[0805] Step 4:

[0806] The server decodes the video data received via the HTTP request and restores the original video frames from the received binary data.

[0807] Step 5:

[0808] The server preprocesses the decoded video frames to make them suitable for input to a pre-trained AI model, specifically by adjusting and normalizing the frame size.

[0809] Step 6:

[0810] The server then inputs the pre-processed video data into the AI ​​model for analysis, which then makes decisions on fouls and other issues based on the video data.

[0811] Step 7:

[0812] The server generates a decision based on the analysis results of the AI ​​model, such as whether to call a foul or continue playing based on the predicted probability.

[0813] Step 8:

[0814] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. The server confirms that the transmission was successful.

[0815] Step 9:

[0816] The terminal receives the judgment result sent from the server and prepares to notify the user based on the received data.

[0817] Step 10:

[0818] The device captures the user's facial expressions and voice data, which are then sent to the emotion engine in real time. The emotion engine analyzes the data to recognize the user's emotional state (e.g., joy, anger, sadness, etc.).

[0819] Step 11:

[0820] The emotion engine determines the user's emotions based on the analysis results, and classifies the results as emotional states such as "normal," "excited," and "tense."

[0821] Step 12:

[0822] The device combines the results of the judgment and the emotion engine to determine the final notification content. For example, if the user is in an "excited" state in the market, the notification content will be adjusted to allow the user to process the situation more calmly.

[0823] Step 13:

[0824] The device will display the final judgement results on its screen and, if necessary, provide an audio notification, allowing users to use this information to ensure fair game management and make appropriate decisions immediately.

[0825] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language during competitions.

[0826] Example 2

[0827] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0828] Judgments made by referees can be heavily subjective and lack fairness. Furthermore, referees face a significant psychological and physical burden, particularly in sports where quick and accurate decisions must be made in real time. Their skills and physical condition can have a significant impact. Furthermore, the issue of a shortage of referees and problems such as violence and verbal abuse during competitions cannot be ignored. To solve these issues, a system is needed that uses video data to automatically make objective and prompt decisions.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0830] In this invention, the server includes means for analyzing video data and generating a judgment result, means for transmitting the generated judgment result to the terminal, and means for analyzing the user's emotions and dynamically adjusting the notification content based on the analysis result, which enables fair and prompt game management, reduces the burden on referees, and improves the quality of sports competitions.

[0831] "Video Data" means video information of a game or competition captured using a camera or other video capture device.

[0832] A "server" is a computer system in a network that receives video data, analyzes it, and generates and transmits a judgment result.

[0833] A "terminal" is a device worn or used by a user (e.g., AI glasses or a smartphone), which receives the judgment results from the server and notifies the user.

[0834] "User" refers to the referees and other competition organizers who use this system and receive judgement results and notifications via their terminals.

[0835] The "emotion engine" is a mechanism that analyzes the user's facial expressions and voice data to recognize the user's emotional state and dynamically adjusts the content of notifications.

[0836] "Real-time" refers to processing or communication that requires minimal delay and immediacy.

[0837] "Encoding" is the process of converting raw data into an efficient format, and in this context refers specifically to the compression of video data using a video codec.

[0838] An "HTTP request" is a communication request based on a protocol for sending and receiving data between a client and a server.

[0839] The "JSON format" is a lightweight data exchange format for structuring and representing data.

[0840] A "generative AI model" is a pre-trained artificial intelligence algorithm that is used to analyze video data and generate judgment results.

[0841] This invention relates to a system that automates the role of referees in sports competitions and supports fair and efficient game management. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to dynamically adjust notification content and user interface.

[0842] The main components of the system are the terminal, the server, and the emotion engine. Specifically, the following hardware and software are used:

[0843] Device: Refers to the AI ​​glasses or smartphone worn by the user, which includes a camera and video processing application for capturing game video data in real time. The device encodes the video data using a video codec such as H.264 and sends the data to the server via an HTTP request.

[0844] Server: A computer system that receives and analyzes video data. The server uses a generative AI model built using machine learning frameworks such as TensorFlow or PyTorch to generate a judgment result from the video data. The judgment result is encoded in JSON format and sent to the device via an HTTP request.

[0845] Emotion engine: A system that analyzes the user's facial expressions and voice to recognize their emotional state. It uses emotion analysis APIs from IBM Watson and Microsoft Azure, and dynamically adjusts notification content based on the results.

[0846] (Example)

[0847] For example, during a soccer match, the device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server then analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device then notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action. The emotion engine also recognizes the user's emotions, and if the user expresses feelings such as impatience or excitement, the notification content is changed to a more specific and appropriate method, helping the user to make quick and appropriate decisions.

[0848] (Example of a prompt)

[0849] "Please explain the system in which a device worn by the user (for example, AI glasses or a smartphone) acquires video data of the game in real time and determines whether or not a foul has been committed. Also, please specify the components of the system that recognizes the user's emotions and adjusts the notification content in accordance with the guidelines."

[0850] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language in competitions.

[0851] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0852] Step 1:

[0853] The device acquires video data of the game in real time. The camera continuously captures frame data and temporarily stores it in memory. Specifically, the device's camera captures video frames every 30 seconds and stores the frame data in memory. The input of this step is the actual video of the game, and the output is the captured frame data.

[0854] Step 2:

[0855] The device encodes the captured video data using the H.264 video codec. The encoding process is performed efficiently using the device's CPU and GPU. Specifically, the captured video frames are compressed and converted into a format suitable for transmission. The input to this step is the captured frame data, and the output is the encoded video data.

[0856] Step 3:

[0857] The device sends the encoded video data to the server via an HTTP request. The HTTP request is generated every 3 seconds and contains the encoded data as a payload. Specifically, the data is temporarily stored in a send buffer and transmitted over a network connection (e.g., Wi-Fi or LTE). The input of this step is the encoded video data, and the output is an HTTP request.

[0858] Step 4:

[0859] The server decodes and preprocesses the received video data. Preprocessing includes noise reduction and resolution adjustment. Specifically, it applies filters to each frame to ensure data quality. The input of this step is the encoded video data sent in the HTTP request, and the output is the preprocessed video data.

[0860] Step 5:

[0861] The server inputs the preprocessed video data into a generative AI model for analysis. This analysis uses pre-trained AI algorithms to determine player movements for each frame. For example, it determines whether contact is a rule violation. The input for this step is the preprocessed video data, and the output is the decision result.

[0862] Step 6:

[0863] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. Specifically, it converts the judgment result into a data structure and generates and sends an HTTP response. The input to this step is the judgment result, and the output is data encoded in JSON format.

[0864] Step 7:

[0865] The device analyzes the received decision result and notifies the user. The notification method is to show the result on the screen or notify the user audibly through the speaker. Specifically, the decision result, such as "foul," is communicated to the user visually and audibly. The input to this step is the decision result in JSON format, and the output is a notification to the user.

[0866] Step 8:

[0867] The device captures the user's facial expressions and voice and sends them to the emotion engine. The emotion engine analyzes facial expressions and voice to recognize the user's emotional state. Specifically, it determines emotions such as impatience or excitement from facial expressions. The input to this step is the user's facial expression and voice data, and the output is the emotion analysis results.

[0868] Step 9:

[0869] The server adjusts the notification content based on the emotion analysis results. For example, if the user is excited, it generates an additional message such as "Please stay calm." The input of this step is the emotion analysis results, and the output is the adjusted notification content.

[0870] Step 10:

[0871] The device re-notifies the user of the adjusted notification content. Specifically, it conveys specific and appropriate instructions to the user by displaying them on the display or outputting them as audio from the speaker. The input of this step is the adjusted notification content, and the output is the notification to the user.

[0872] The above processing steps enable fair and efficient game management.

[0873] (Application example 2)

[0874] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0875] In today's manufacturing workplaces, product quality control and monitoring of workers' emotional states are highly valued, but performing these tasks efficiently can be difficult. Quality inspections and evaluations of workers' emotional states, in particular, rely on human judgment and involve subjective factors, which can lead to a lack of fairness and accuracy. Furthermore, properly understanding workers' fatigue and stress and encouraging them to take necessary breaks is important from the perspective of improving productivity and preventing accidents. Therefore, there is a need for fair and accurate quality control and the establishment of a system that can monitor and evaluate workers' emotional states in real time.

[0876] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data to the server, means for the server to analyze the video data and generate a determination result, means for transmitting the generated determination result to the terminal, means for notifying the user of the determination result received by the terminal, means for dynamically adjusting the notification content using an emotion engine that recognizes the user's emotions, means for performing quality control in a factory environment using an artificial intelligence model, and means for analyzing the video data based on prompt sentences to determine the emotions of workers. This enables fair and accurate quality control, and also makes it possible to grasp the emotional state of workers in real time and provide appropriate notifications or interventions as needed.

[0877] "Video data" refers to information about moving images acquired using devices such as cameras and sensors.

[0878] A "server" is a computer system that transmits, receives, and processes data over a network.

[0879] A "terminal" is a device that is operated by a user and has the function of displaying or notifying received information.

[0880] An "emotion engine" is an artificial intelligence technology that analyzes data such as a user's facial expressions and voice to recognize their emotional state in real time.

[0881] An "artificial intelligence model" is an algorithm that learns from large amounts of data and performs specific tasks.

[0882] "Quality control" is a set of activities and processes for maintaining the quality of products and services to a certain standard.

[0883] A "generative AI model" is an artificial intelligence model that uses techniques such as generative adversarial networks (GANs) and natural language processing (NLP) to generate new data and text.

[0884] A "prompt sentence" is an instruction sentence input to a generative AI model that determines the direction of the output generated by the model.

[0885] The system for implementing this invention mainly comprises a set of means for acquiring video data, analyzing it, and generating and notifying results. It also utilizes an emotion engine and artificial intelligence model to monitor quality control and the emotional state of workers in a factory environment.

[0886] System Configuration

[0887] 1. Acquiring video data

[0888] The device uses a camera to capture video data in real time. This video data can be used for, for example, product quality inspections or monitoring the work environment. The camera can be a general webcam or an industrial camera.

[0889] 2. Data transmission

[0890] The device encodes the acquired video data into an appropriate format and sends it to the server via an HTTP request, using a library such as the requests library.

[0891] 3. Analysis of video data

[0892] The server decodes the received video data and analyzes it using a pre-trained generative AI model. Specifically, it detects defects for quality control and determines the emotional state of workers. This analysis uses machine learning libraries such as PyTorch and TensorFlow.

[0893] 4. Generating and sending analysis results

[0894] The server encodes the analysis results in JSON format and sends them to the terminal via an HTTP request. The analysis results include information such as "poor quality" and "worker stress level."

[0895] 5. Result Notification and Emotion Engine

[0896] The device then notifies the user of the received judgment results by displaying a message on the screen or issuing a voice alert. The device also utilizes an emotion engine that recognizes the user's emotions in real time and dynamically adjusts the content of the notification based on the analysis results.

[0897] Specific examples

[0898] For example, consider a worker wearing smart glasses inspecting products. The system detects defective products in real time and displays a message to the worker, such as "A defective product has been detected." If the emotion engine detects stress in the worker, it will send an appropriate notification, such as "You are feeling stressed. We recommend that you take a short break." This enables fair and accurate quality control while improving worker safety and work efficiency.

[0899] Specific hardware and software used

[0900] Camera: General webcam, industrial camera

[0901] Devices: Smartphones, smart glasses, etc.

[0902] Server: High-performance computing server (e.g. AWS EC2, Google Cloud Platform)

[0903] Libraries: requests (data transmission), cv2 (video capture), PyTorch or TensorFlow (AI model analysis)

[0904] Prompt Sentence Examples

[0905] Analyze input images to determine if they contain defects, and analyze the user's emotional state to recognize stress or fatigue and provide appropriate notifications accordingly.

[0906] This invention significantly improves work efficiency and quality control at manufacturing sites and reduces the burden on workers.

[0907] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0908] Step 1:

[0909] The device acquires video data in real time using a camera. Specifically, it continuously captures frame data from the camera device. The acquired video data is input in frame format (e.g., JPEG).

[0910] Step 2:

[0911] The video data acquired by the device is encoded into an appropriate format (e.g., JPEG). The encoded data is then processed into a format that is easy to send, and becomes output data to be sent to the server in the form of an HTTP request. This process is performed using cv2.imencode or similar.

[0912] Step 3:

[0913] The device sends the encoded video data to the server using an HTTP request. Specifically, it sends the encoded frame data to the server using a method such as requests.post. The input is the encoded video data, and the output is the result of the data transmission to the server.

[0914] Step 4:

[0915] The server decodes the video data it receives. Specifically, it uses functions such as cv2.imdecode on the server side to restore the encoded data back to frame data. The input is the encoded video data, and the output is the decoded frame data.

[0916] Step 5:

[0917] The server analyzes the received video data using a generative AI model. Specifically, using libraries such as PyTorch and TensorFlow, the frame data is input into the AI ​​model to generate a judgment result. The input here is the decoded video data, and the output is the analysis result (e.g., whether there is a quality problem or the emotional state).

[0918] Step 6:

[0919] The server encodes the analysis results into JSON format and sends them to the terminal via an HTTP request. Specifically, it encodes the analysis results into JSON format using json.dumps or similar and sends them via an HTTP request. The input is the analysis results and the output is JSON format data.

[0920] Step 7:

[0921] The device notifies the user of the result of the judgment received. Specifically, it displays a message on the screen or uses an audio alert. The input is the analysis result in JSON format, and the output is a notification to the user (e.g., text message, audio message).

[0922] Step 8:

[0923] The device uses an emotion engine that recognizes the user's emotions in real time. Specifically, the user's facial expressions and voice data are input into the emotion engine, which analyzes their emotional state. The input is the user's facial expressions and voice data, and the output is the emotion analysis results.

[0924] Step 9:

[0925] The device dynamically adjusts the notification content based on the emotion analysis results. For example, if the user is under stress, the notification message will be changed to "We recommend you take a break." The input is the emotion analysis results, and the output is the adjusted notification content.

[0926] Through these steps, the system achieves fair and accurate quality control and real-time monitoring and evaluation of workers' emotional states.

[0927] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0928] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0929] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0930] [Fourth embodiment]

[0931] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0932] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0933] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0934] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0935] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0936] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0937] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0938] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0939] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0940] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0941] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0942] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0943] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0944] The present invention provides a system for automating the role of referees in sports competitions and supporting fair and efficient game management. This system has a series of means for acquiring video data in real time, transmitting the acquired video data to a server, analyzing the received video data and generating a judgment result, transmitting the generated judgment result to a terminal, and notifying the user of the judgment result received by the terminal.

[0945] (System configuration)

[0946] 1. Acquiring video data

[0947] The device (such as a smartphone or AI glasses worn by the user) acquires video data of the game in real time. It uses a camera to continuously capture frames of data and processes them in real-time streaming.

[0948] 2. Transmission of video data

[0949] The device sends the acquired video data to the server, where it performs encoding and sends the encoded data to the server via an HTTP request.

[0950] 3. Analysis of video data

[0951] The server analyzes the received video data and generates a decision. A pre-trained AI model is used for the analysis, and the video data is pre-processed before being input into the AI ​​model. The decision results include whether or not a foul was committed and the scoring situation.

[0952] 4. Sending the results

[0953] The server sends the generated judgment result to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request.

[0954] 5. Notification of the decision

[0955] The device notifies the user of the received judgment result, which may be displayed on a screen or output as audio.

[0956] (Example)

[0957] For example, during a soccer match, a device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action.

[0958] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language during competitions.

[0959] The processing flow will be explained below.

[0960] Step 1:

[0961] The device initializes the camera device and starts video capture. The device uses the camera to capture successive frames in real time and generate a video stream.

[0962] Step 2:

[0963] The device encodes the received frame data into an appropriate format, such as JPEG, and processes the frame as binary data that can be transmitted.

[0964] Step 3:

[0965] The device sends the encoded video data to the server via an HTTP request. The device checks the status of the transmission to see if it was successful or not, and attempts to retransmit if necessary.

[0966] Step 4:

[0967] The server decodes the video data received via the HTTP request and restores the original video frames from the received binary data.

[0968] Step 5:

[0969] The server preprocesses the decoded video frames to make them suitable for input to a pre-trained AI model, specifically by adjusting and normalizing the frame size.

[0970] Step 6:

[0971] The server then inputs the pre-processed video data into the AI ​​model for analysis, which then makes decisions on fouls and other issues based on the video data.

[0972] Step 7:

[0973] The server generates a decision based on the analysis results of the AI ​​model, such as whether to call a foul or continue playing based on the predicted probability.

[0974] Step 8:

[0975] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. The server confirms that the transmission was successful.

[0976] Step 9:

[0977] The terminal receives the judgment result sent from the server and prepares to notify the user based on the received data.

[0978] Step 10:

[0979] The device then notifies the user of the received decision results. Possible notification methods include displaying the results on a screen or announcing the results by voice, allowing the user to check the results of the match in real time.

[0980] Example 1

[0981] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0982] Fair and prompt refereeing decisions are important in modern sports competitions. However, problems such as human error and a shortage of referees can have a significant impact on the management of games. Conventional technology has not sufficiently reduced the burden on referees or improved the accuracy of their decisions, so a new system is needed to improve the quality of sports competitions.

[0983] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0984] In this invention, the server includes means for decoding received video data and analyzing it using an AI model, means for encoding the analysis results in JSON format and transmitting them to the terminal as judgment results, and means for notifying the user of the judgment results received by the terminal. This enables fair and prompt judgment, reduces the burden on referees, and improves the quality of sports competitions.

[0985] "Video data" is a collection of video frames captured during a sporting event.

[0986] "Preprocessing" refers to processing such as noise removal and frame rate adjustment that is carried out before analyzing video data.

[0987] "Encoding" is the process of compressing and converting video data into a specific format.

[0988] A "server" is a computer system that receives video data, decodes it, and analyzes it using an AI model.

[0989] "Decoding" is the process of restoring encoded data to its original form.

[0990] An "AI model" is a computational model that uses pre-trained algorithms to analyze video data.

[0991] "Analysis" is the process of using an AI model to evaluate the content of video data and derive a judgment result.

[0992] The "decision result" is judgment information obtained as a result of analysis, and indicates, for example, whether or not a foul has been committed.

[0993] The "JSON format" is a lightweight data exchange format for structuring and expressing data such as judgment results.

[0994] "Notification" refers to the action of the terminal notifying the user of the determination result.

[0995] A "terminal" is a device worn or used by a user, and is an electronic device that displays the received judgment results or notifies them by voice.

[0996] The present invention provides a system for automating the role of referees in sports competitions and supporting fair and efficient game management. This system has a series of means for acquiring video data in real time, transmitting the acquired video data to a server, analyzing the received video data and generating a judgment result, transmitting the generated judgment result to a terminal, and notifying the user of the judgment result received by the terminal.

[0997] Specifically, the AI ​​glasses worn by the user or a device such as a smartphone acquires video data during the match in real time. For example, the device's camera continuously captures frame data, which is then processed using real-time streaming technology (such as the RTSP protocol).

[0998] The device preprocesses the captured video data and then performs encoding. Preprocessing involves removing noise from the video data and adjusting the frame rate, for example, converting the video data resolution to 720p. Next, the video data is encoded (for example, using the H.264 codec) and compressed. This encoded data is then sent to the server using an HTTP request.

[0999] The server decodes the received video data and inputs it into a pre-trained AI model. For analysis, it uses machine learning frameworks such as TensorFlow and PyTorch. The server uses the AI ​​model to extract features for each frame and analyze and detect contact between players and rule violations (fouls).

[1000] The result of the analysis, such as a "foul" judgment, is encoded in JSON format and sent to the terminal via an HTTP request.

[1001] The device will then notify the user of the received decision. Specifically, it may display "Foul" on the device's display or announce "A foul has been detected" through a voice notification. The device may also flash a portion of the screen in red as visual feedback to the user.

[1002] Examples:

[1003] For example, imagine a referee wearing AI glasses during a soccer match. These AI glasses capture video of the match in real time. The captured video data undergoes noise reduction and frame rate adjustment on the device, and is then encoded using the H.264 codec. The encoded data is sent to the server via an HTTP POST request. The server decodes the video data and uses an AI model (e.g., ResNet-50) to analyze the player movements and contact on a frame-by-frame basis. If the analysis determines that contact between players constitutes a rules violation (a foul), the server encodes that information into JSON format and sends it back to the device via an HTTP POST request.

[1004] When the device receives this decision, it will display "Foul" on the screen and announce "A foul has been detected" via voice, allowing the referee (user) to take appropriate action immediately.

[1005] Example prompt sentence:

[1006] "During a soccer match, the AI ​​glasses worn by the user will capture video of the match in real time and send that video data to a server. The server will analyze the video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis result indicates a foul, the decision will be sent to the AI ​​glasses, which will then notify the user (the referee) that a 'foul' has occurred. Please generate a program for this process."

[1007] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1008] Step 1:

[1009] The device acquires video data. Specifically, AI glasses worn by the user or a smartphone camera captures video data of the game in real time. The input is video of the game, and the output is continuous frame data. For example, the device captures video at 30 frames per second. The captured video data is then sent to the next step.

[1010] Step 2:

[1011] The video data captured by the device is preprocessed and then encoded. Preprocessing involves removing noise from the video data and adjusting the frame rate. For example, converting the resolution of the video data to 720p. The input is the captured raw video data, and the output is preprocessed video data. The preprocessed video data is then encoded using the H.264 codec. The input is the preprocessed video data, and the output is encoded compressed video data.

[1012] Step 3:

[1013] The device sends the encoded video data to the server via an HTTP request. The input is the encoded data, and the output is an HTTP request to the server. Specifically, the device sends the following HTTP POST request to the server:

[1014] POST / api / upload HTTP / 1.1

[1015] Host: server.example.com

[1016] Content-Type: application / octet-stream

[1017] Content-Length: [data length]

[1018] [data]

[1019] Step 4:

[1020] The server decodes the received video data and inputs it into the AI ​​model. First, the server decodes the encoded data to restore the original video data. The encoded data is the input, and the decoded video data is the output. Next, the server inputs the decoded video data into a pre-trained AI model (using, for example, TensorFlow or PyTorch) and performs analysis. The decoded video data is the input, and the analysis results are the output.

[1021] Step 5:

[1022] The server generates a verdict based on the analysis results of the AI ​​model and encodes it in JSON format. Specifically, the information obtained as a result of the analysis (such as whether or not there was a foul) is encoded in JSON format. The input is the analysis results of the AI ​​model, and the output is the verdict encoded in JSON format. For example, the JSON data of the verdict looks like this:

[1023] json

[1024] {

[1025] "event": "foul",

[1026] "timestamp": "2023-10-01T15:04:05Z",

[1027] "details": {

[1028] "player": "Player 10",

[1029] "team": "Team A"

[1030] }

[1031] }

[1032] Step 6:

[1033] The server generates a judgment result and sends it to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request. The input is the judgment result in JSON format, and the output is an HTTP request to the terminal. Specifically, the server sends the following HTTP POST request to the terminal:

[1034] POST / api / notification HTTP / 1.1

[1035] Host: terminal.example.com

[1036] Content-Type: application / json

[1037] Content-Length: [data length]

[1038] [JSON data]

[1039] Step 7:

[1040] The device notifies the user of the decision result received. Specifically, it displays "Foul" on the device's display or announces "A foul has been detected" through an audio notification. The input is the decision result received from the server, and the output is a visual or audio notification to the user. The device may also flash a part of the screen red as visual feedback to the user.

[1041] (Application example 1)

[1042] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1043] Conventional security monitoring systems have difficulty detecting suspicious behavior or individuals in real time, and consume a large amount of human resources, especially in environments that require wide-area monitoring. Visual monitoring by on-site security personnel is inefficient and can result in delays in situations that require immediate response. Furthermore, the accuracy of detecting suspicious behavior depends on the viewer's subjective judgment, leading to inconsistent judgments. There is a need to solve these issues and improve the efficiency and accuracy of security monitoring.

[1044] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1045] In this invention, the server includes a means for analyzing video data and generating a judgment result, a means for using a generative AI model to detect suspicious behaviors and people in real time, and a means for executing the AI ​​model based on a prompt sentence, thereby enabling highly accurate detection of suspicious behaviors and people in real time.

[1046] "Video data" is a collection of visual information generated from a camera or other image capture device.

[1047] A "server" is an information processing device or system that receives and transmits data via a network and performs processes such as analysis and storage.

[1048] A "terminal" is a device that can be directly operated by a user and that transmits and receives data to and from a server. For example, a smartphone or smart glasses fall into this category.

[1049] A "generative AI model" is a model that uses pre-trained artificial intelligence and is capable of performing specified tasks.

[1050] A "prompt" is an instruction or input text that an AI model uses to perform a specific task, guiding the AI ​​to take appropriate action.

[1051] "Real time" means that processing is carried out the moment an event occurs, and indicates a situation in which an immediate response can be made without delay.

[1052] "Suspicious behavior" refers to abnormal behavior that deviates from normal patterns of behavior and may pose a security risk.

[1053] "Person" refers to a human being within a monitored area who is the subject of security assessment and behavior monitoring.

[1054] An "HTTP request" is a protocol used to send and receive data between a web server and a client, and is a means for making a request to a server.

[1055] "Encoding" is the process of converting data into a particular format, and in the case of video data, this means converting it into a digital format that is easier to transmit or store.

[1056] "Decoding" refers to the process of restoring encoded data to its original form, meaning converting the received data into an analyzable format.

[1057] This invention is a system that automates security monitoring and realizes highly accurate detection of suspicious behavior and individuals. This system executes a series of processes: acquisition, transmission, analysis, generation of judgment results, and notification of video data. Each component of the system and its function are described below.

[1058] First, the user's device (e.g., a smartphone or smart glasses) acquires video data in real time from a surveillance camera or the device's own camera. The acquired video data is encoded into an appropriate format (e.g., JPEG) and sent to the server via an HTTP request. This is done using OpenCV and the requests library.

[1059] The server decodes the transmitted video data and analyzes the video using a pre-trained generative AI model (e.g., a model using TensorFlow Keras). This generative AI model is designed to detect suspicious behavior and people with high accuracy. As a result of the analysis, a list of suspicious behavior and people is generated.

[1060] The analysis results are encoded in JSON format and sent to the device again via an HTTP request. The device then notifies the user of the received judgment results. Possible notification methods include displaying the results on a screen or outputting audio. This process allows the user to receive real-time detection results for suspicious behavior and people.

[1061] As a concrete example, consider a surveillance system in a commercial facility. This system analyzes video footage from multiple surveillance cameras in real time to detect theft and suspicious activity. For example, the following prompts could be used:

[1062] "Detect suspicious behavior or people from video data. The subject is surveillance footage from a commercial facility."

[1063] Based on this prompt, the generative AI model performs a specific task and notifies the device of the results.

[1064] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1065] Step 1:

[1066] The device acquires the video data

[1067] The device (smartphone or smart glasses) captures video data in real time using a surveillance camera or built-in camera. The input is the subject of the video effect, and the output is a series of frames of video data. Specifically, it monitors an area specified by the user and captures frames at specified intervals.

[1068] Step 2:

[1069] The device sends the video data to the server

[1070] The captured video data is encoded (e.g., into JPEG format) and sent to the server via an HTTP request. The input is the captured video frame, and the output is the encoded data. Specifically, the video is temporarily stored on the device and then sent to the server via an appropriate network connection.

[1071] Step 3:

[1072] The server receives and decodes the video data

[1073] The server receives the video data sent from the device and performs the decoding process. The input is encoded video data, and the output is video data in an analyzable format. Specifically, the server receives an HTTP request, saves the received data in memory, and uses a decoding library to restore the original video data.

[1074] Step 4:

[1075] The server analyzes the video data based on the generated AI model

[1076] The server inputs the decoded video data into a generative AI model for analysis. The input is the decoded video data, and the output is the detection results of suspicious behavior and people. Specifically, the server uses AI libraries such as TensorFlow and Keras to analyze the input video in real time and execute the generative AI model according to the prompt text.

[1077] Step 5:

[1078] The server sends the analysis results to the terminal in JSON format.

[1079] The server encodes the analysis results in JSON format and sends them to the terminal via an HTTP request. The input is the analysis results, and the output is JSON format data. Specifically, the server encodes the analysis results generated by the server using a JSON library and sends an HTTP request to the address specified by the terminal.

[1080] Step 6:

[1081] The device receives the analysis results and notifies the user.

[1082] The device receives the analysis results in JSON format sent from the server and notifies the user of the contents. The input is the analysis results in JSON format, and the output is a notification to the user (display or audio output). Specifically, the device analyzes the received data and notifies the user of the results of the detection of suspicious behavior or people via the screen display or audio output device.

[1083] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1084] This invention relates to a system that automates the role of referees in sports competitions and supports fair and efficient game management. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to dynamically adjust notification content and user interface.

[1085] (System configuration)

[1086] 1. Acquiring video data

[1087] The device (such as a smartphone or AI glasses worn by the user) acquires video data of the game in real time. It uses a camera to continuously capture frames of data and processes them in real-time streaming.

[1088] 2. Transmission of video data

[1089] The device sends the acquired video data to the server, where it performs encoding and sends the encoded data to the server via an HTTP request.

[1090] 3. Analysis of video data

[1091] The server analyzes the received video data and generates a decision. A pre-trained AI model is used for the analysis, and the video data is pre-processed before being input into the AI ​​model. The decision results include whether or not a foul was committed and the scoring situation.

[1092] 4. Sending the results

[1093] The server sends the generated judgment result to the terminal. The judgment result is encoded in JSON format and sent to the terminal via an HTTP request.

[1094] 5. Notification of the decision

[1095] The device notifies the user of the received judgment result, which may be displayed on a screen or output as audio.

[1096] 6. Emotion Engine

[1097] The emotion engine has a means for recognizing the user's emotions in real time. The device captures the user's facial expressions and voice and sends the data to the emotion engine. The emotion engine analyzes the user's emotional state and adjusts the notification content based on the analysis results.

[1098] (Example)

[1099] For example, during a soccer match, a device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action.

[1100] In addition, the emotion engine recognizes the user's emotions, and if the user expresses, for example, impatience or excitement, the notification content is changed to a more specific and appropriate method, helping the user make quick and appropriate decisions. This system enables fair and prompt judgment, reduces the burden on referees, and improves the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language in competitions.

[1101] The processing flow will be explained below.

[1102] Step 1:

[1103] The device initializes the camera device and starts video capture. The device uses the camera to capture successive frames in real time and generate a video stream.

[1104] Step 2:

[1105] The device encodes the received frame data into an appropriate format, such as JPEG, and processes the frame as binary data that can be transmitted.

[1106] Step 3:

[1107] The device sends the encoded video data to the server via an HTTP request. The device checks the status of the transmission to see if it was successful or not, and attempts to retransmit if necessary.

[1108] Step 4:

[1109] The server decodes the video data received via the HTTP request and restores the original video frames from the received binary data.

[1110] Step 5:

[1111] The server preprocesses the decoded video frames to make them suitable for input to a pre-trained AI model, specifically by adjusting and normalizing the frame size.

[1112] Step 6:

[1113] The server then inputs the pre-processed video data into the AI ​​model for analysis, which then makes decisions on fouls and other issues based on the video data.

[1114] Step 7:

[1115] The server generates a decision based on the analysis results of the AI ​​model, such as whether to call a foul or continue playing based on the predicted probability.

[1116] Step 8:

[1117] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. The server confirms that the transmission was successful.

[1118] Step 9:

[1119] The terminal receives the judgment result sent from the server and prepares to notify the user based on the received data.

[1120] Step 10:

[1121] The device captures the user's facial expressions and voice data, which are then sent to the emotion engine in real time. The emotion engine analyzes the data to recognize the user's emotional state (e.g., joy, anger, sadness, etc.).

[1122] Step 11:

[1123] The emotion engine determines the user's emotions based on the analysis results, and classifies the results as emotional states such as "normal," "excited," and "tense."

[1124] Step 12:

[1125] The device combines the results of the judgment and the emotion engine to determine the final notification content. For example, if the user is in an "excited" state in the market, the notification content will be adjusted to allow the user to process the situation more calmly.

[1126] Step 13:

[1127] The device will display the final judgement results on its screen and, if necessary, provide an audio notification, allowing users to use this information to ensure fair game management and make appropriate decisions immediately.

[1128] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language during competitions.

[1129] Example 2

[1130] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1131] Judgments made by referees can be heavily subjective and lack fairness. Furthermore, referees face a significant psychological and physical burden, particularly in sports where quick and accurate decisions must be made in real time. Their skills and physical condition can have a significant impact. Furthermore, the issue of a shortage of referees and problems such as violence and verbal abuse during competitions cannot be ignored. To solve these issues, a system is needed that uses video data to automatically make objective and prompt decisions.

[1132] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1133] In this invention, the server includes means for analyzing video data and generating a judgment result, means for transmitting the generated judgment result to the terminal, and means for analyzing the user's emotions and dynamically adjusting the notification content based on the analysis result, which enables fair and prompt game management, reduces the burden on referees, and improves the quality of sports competitions.

[1134] "Video Data" means video information of a game or competition captured using a camera or other video capture device.

[1135] A "server" is a computer system in a network that receives video data, analyzes it, and generates and transmits a judgment result.

[1136] A "terminal" is a device worn or used by a user (e.g., AI glasses or a smartphone), which receives the judgment results from the server and notifies the user.

[1137] "User" refers to the referees and other competition organizers who use this system and receive judgement results and notifications via their terminals.

[1138] The "emotion engine" is a mechanism that analyzes the user's facial expressions and voice data to recognize the user's emotional state and dynamically adjusts the content of notifications.

[1139] "Real-time" refers to processing or communication that requires minimal delay and immediacy.

[1140] "Encoding" is the process of converting raw data into an efficient format, and in this context refers specifically to the compression of video data using a video codec.

[1141] An "HTTP request" is a communication request based on a protocol for sending and receiving data between a client and a server.

[1142] The "JSON format" is a lightweight data exchange format for structuring and representing data.

[1143] A "generative AI model" is a pre-trained artificial intelligence algorithm that is used to analyze video data and generate judgment results.

[1144] This invention relates to a system that automates the role of referees in sports competitions and supports fair and efficient game management. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to dynamically adjust notification content and user interface.

[1145] The main components of the system are the terminal, the server, and the emotion engine. Specifically, the following hardware and software are used:

[1146] Device: Refers to the AI ​​glasses or smartphone worn by the user, which includes a camera and video processing application for capturing game video data in real time. The device encodes the video data using a video codec such as H.264 and sends the data to the server via an HTTP request.

[1147] Server: A computer system that receives and analyzes video data. The server uses a generative AI model built using machine learning frameworks such as TensorFlow or PyTorch to generate a judgment result from the video data. The judgment result is encoded in JSON format and sent to the device via an HTTP request.

[1148] Emotion engine: A system that analyzes the user's facial expressions and voice to recognize their emotional state. It uses emotion analysis APIs from IBM Watson and Microsoft Azure, and dynamically adjusts notification content based on the results.

[1149] (Example)

[1150] For example, during a soccer match, the device (AI glasses worn by the user) automatically captures video data. This video data is sent to a server in real time. The server then analyzes the received video data using an AI model to determine whether contact between players constitutes a rules violation (foul). If the analysis results in a "foul," the server sends the result to the device. The device then notifies the user of this determination, allowing the user (the referee) to immediately take appropriate action. The emotion engine also recognizes the user's emotions, and if the user expresses feelings such as impatience or excitement, the notification content is changed to a more specific and appropriate method, helping the user to make quick and appropriate decisions.

[1151] (Example of a prompt)

[1152] "Please explain the system in which a device worn by the user (for example, AI glasses or a smartphone) acquires video data of the game in real time and determines whether or not a foul has been committed. Also, please specify the components of the system that recognizes the user's emotions and adjusts the notification content in accordance with the guidelines."

[1153] This system will enable fair and prompt judgment, reduce the burden on referees, and improve the quality of sports competitions. It is also expected to solve the problem of referee shortages and reduce the risk of violence and abusive language in competitions.

[1154] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1155] Step 1:

[1156] The device acquires video data of the game in real time. The camera continuously captures frame data and temporarily stores it in memory. Specifically, the device's camera captures video frames every 30 seconds and stores the frame data in memory. The input of this step is the actual video of the game, and the output is the captured frame data.

[1157] Step 2:

[1158] The device encodes the captured video data using the H.264 video codec. The encoding process is performed efficiently using the device's CPU and GPU. Specifically, the captured video frames are compressed and converted into a format suitable for transmission. The input to this step is the captured frame data, and the output is the encoded video data.

[1159] Step 3:

[1160] The device sends the encoded video data to the server via an HTTP request. The HTTP request is generated every 3 seconds and contains the encoded data as a payload. Specifically, the data is temporarily stored in a send buffer and transmitted over a network connection (e.g., Wi-Fi or LTE). The input of this step is the encoded video data, and the output is an HTTP request.

[1161] Step 4:

[1162] The server decodes and preprocesses the received video data. Preprocessing includes noise reduction and resolution adjustment. Specifically, it applies filters to each frame to ensure data quality. The input of this step is the encoded video data sent in the HTTP request, and the output is the preprocessed video data.

[1163] Step 5:

[1164] The server inputs the preprocessed video data into a generative AI model for analysis. This analysis uses pre-trained AI algorithms to determine player movements for each frame. For example, it determines whether contact is a rule violation. The input for this step is the preprocessed video data, and the output is the decision result.

[1165] Step 6:

[1166] The server encodes the generated judgment result in JSON format and sends it to the terminal via an HTTP request. Specifically, it converts the judgment result into a data structure and generates and sends an HTTP response. The input to this step is the judgment result, and the output is data encoded in JSON format.

[1167] Step 7:

[1168] The device analyzes the received decision result and notifies the user. The notification method is to show the result on the screen or notify the user audibly through the speaker. Specifically, the decision result, such as "foul," is communicated to the user visually and audibly. The input to this step is the decision result in JSON format, and the output is a notification to the user.

[1169] Step 8:

[1170] The device captures the user's facial expressions and voice and sends them to the emotion engine. The emotion engine analyzes facial expressions and voice to recognize the user's emotional state. Specifically, it determines emotions such as impatience or excitement from facial expressions. The input to this step is the user's facial expression and voice data, and the output is the emotion analysis results.

[1171] Step 9:

[1172] The server adjusts the notification content based on the emotion analysis results. For example, if the user is excited, it generates an additional message such as "Please stay calm." The input of this step is the emotion analysis results, and the output is the adjusted notification content.

[1173] Step 10:

[1174] The device re-notifies the user of the adjusted notification content. Specifically, it conveys specific and appropriate instructions to the user by displaying them on the display or outputting them as audio from the speaker. The input of this step is the adjusted notification content, and the output is the notification to the user.

[1175] The above processing steps enable fair and efficient game management.

[1176] (Application example 2)

[1177] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1178] In today's manufacturing workplaces, product quality control and monitoring of workers' emotional states are highly valued, but performing these tasks efficiently can be difficult. Quality inspections and evaluations of workers' emotional states, in particular, rely on human judgment and involve subjective factors, which can lead to a lack of fairness and accuracy. Furthermore, properly understanding workers' fatigue and stress and encouraging them to take necessary breaks is important from the perspective of improving productivity and preventing accidents. Therefore, there is a need for fair and accurate quality control and the establishment of a system that can monitor and evaluate workers' emotional states in real time.

[1179] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring video data, means for transmitting the acquired video data to the server, means for the server to analyze the video data and generate a determination result, means for transmitting the generated determination result to the terminal, means for notifying the user of the determination result received by the terminal, means for dynamically adjusting the notification content using an emotion engine that recognizes the user's emotions, means for performing quality control in a factory environment using an artificial intelligence model, and means for analyzing the video data based on prompt sentences to determine the emotions of workers. This enables fair and accurate quality control, and also makes it possible to grasp the emotional state of workers in real time and provide appropriate notifications or interventions as needed.

[1180] "Video data" refers to information about moving images acquired using devices such as cameras and sensors.

[1181] A "server" is a computer system that transmits, receives, and processes data over a network.

[1182] A "terminal" is a device that is operated by a user and has the function of displaying or notifying received information.

[1183] An "emotion engine" is an artificial intelligence technology that analyzes data such as a user's facial expressions and voice to recognize their emotional state in real time.

[1184] An "artificial intelligence model" is an algorithm that learns from large amounts of data and performs specific tasks.

[1185] "Quality control" is a set of activities and processes for maintaining the quality of products and services to a certain standard.

[1186] A "generative AI model" is an artificial intelligence model that uses techniques such as generative adversarial networks (GANs) and natural language processing (NLP) to generate new data and text.

[1187] A "prompt sentence" is an instruction sentence input to a generative AI model that determines the direction of the output generated by the model.

[1188] The system for implementing this invention mainly comprises a set of means for acquiring video data, analyzing it, and generating and notifying results. It also utilizes an emotion engine and artificial intelligence model to monitor quality control and the emotional state of workers in a factory environment.

[1189] System Configuration

[1190] 1. Acquiring video data

[1191] The device uses a camera to capture video data in real time. This video data can be used for, for example, product quality inspections or monitoring the work environment. The camera can be a general webcam or an industrial camera.

[1192] 2. Data transmission

[1193] The device encodes the acquired video data into an appropriate format and sends it to the server via an HTTP request, using a library such as the requests library.

[1194] 3. Analysis of video data

[1195] The server decodes the received video data and analyzes it using a pre-trained generative AI model. Specifically, it detects defects for quality control and determines the emotional state of workers. This analysis uses machine learning libraries such as PyTorch and TensorFlow.

[1196] 4. Generating and sending analysis results

[1197] The server encodes the analysis results in JSON format and sends them to the terminal via an HTTP request. The analysis results include information such as "poor quality" and "worker stress level."

[1198] 5. Result Notification and Emotion Engine

[1199] The device then notifies the user of the received judgment results by displaying a message on the screen or issuing a voice alert. The device also utilizes an emotion engine that recognizes the user's emotions in real time and dynamically adjusts the content of the notification based on the analysis results.

[1200] Specific examples

[1201] For example, consider a worker wearing smart glasses inspecting products. The system detects defective products in real time and displays a message to the worker, such as "A defective product has been detected." If the emotion engine detects stress in the worker, it will send an appropriate notification, such as "You are feeling stressed. We recommend that you take a short break." This enables fair and accurate quality control while improving worker safety and work efficiency.

[1202] Specific hardware and software used

[1203] Camera: General webcam, industrial camera

[1204] Devices: Smartphones, smart glasses, etc.

[1205] Server: High-performance computing server (e.g. AWS EC2, Google Cloud Platform)

[1206] Libraries: requests (data transmission), cv2 (video capture), PyTorch or TensorFlow (AI model analysis)

[1207] Prompt Sentence Examples

[1208] Analyze input images to determine if they contain defects, and analyze the user's emotional state to recognize stress or fatigue and provide appropriate notifications accordingly.

[1209] This invention significantly improves work efficiency and quality control at manufacturing sites and reduces the burden on workers.

[1210] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1211] Step 1:

[1212] The device acquires video data in real time using a camera. Specifically, it continuously captures frame data from the camera device. The acquired video data is input in frame format (e.g., JPEG).

[1213] Step 2:

[1214] The video data acquired by the device is encoded into an appropriate format (e.g., JPEG). The encoded data is then processed into a format that is easy to send, and becomes output data to be sent to the server in the form of an HTTP request. This process is performed using cv2.imencode or similar.

[1215] Step 3:

[1216] The device sends the encoded video data to the server using an HTTP request. Specifically, it sends the encoded frame data to the server using a method such as requests.post. The input is the encoded video data, and the output is the result of the data transmission to the server.

[1217] Step 4:

[1218] The server decodes the video data it receives. Specifically, it uses functions such as cv2.imdecode on the server side to restore the encoded data back to frame data. The input is the encoded video data, and the output is the decoded frame data.

[1219] Step 5:

[1220] The server analyzes the received video data using a generative AI model. Specifically, using libraries such as PyTorch and TensorFlow, the frame data is input into the AI ​​model to generate a judgment result. The input here is the decoded video data, and the output is the analysis result (e.g., whether there is a quality problem or the emotional state).

[1221] Step 6:

[1222] The server encodes the analysis results into JSON format and sends them to the terminal via an HTTP request. Specifically, it encodes the analysis results into JSON format using json.dumps or similar and sends them via an HTTP request. The input is the analysis results and the output is JSON format data.

[1223] Step 7:

[1224] The device notifies the user of the result of the judgment received. Specifically, it displays a message on the screen or uses an audio alert. The input is the analysis result in JSON format, and the output is a notification to the user (e.g., text message, audio message).

[1225] Step 8:

[1226] The device uses an emotion engine that recognizes the user's emotions in real time. Specifically, the user's facial expressions and voice data are input into the emotion engine, which analyzes their emotional state. The input is the user's facial expressions and voice data, and the output is the emotion analysis results.

[1227] Step 9:

[1228] The device dynamically adjusts the notification content based on the emotion analysis results. For example, if the user is under stress, the notification message will be changed to "We recommend you take a break." The input is the emotion analysis results, and the output is the adjusted notification content.

[1229] Through these steps, the system achieves fair and accurate quality control and real-time monitoring and evaluation of workers' emotional states.

[1230] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1231] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1232] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1233] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1234] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1235] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1236] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1237] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1238] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1239] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1240] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1241] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1242] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1243] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1244] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1245] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1246] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1247] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1248] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1249] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1250] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1251] The following is further disclosed regarding the above embodiment.

[1252] (Claim 1)

[1253] a means for acquiring video data;

[1254] means for transmitting the acquired video data to a server;

[1255] A means for the server to analyze the video data and generate a determination result;

[1256] means for transmitting the generated determination result to a terminal;

[1257] means for notifying a user of the determination result received by the terminal;

[1258] A system including:

[1259] (Claim 2)

[1260] a means for capturing video data in real time;

[1261] means for encoding the captured video data into an appropriate format;

[1262] a means for transmitting the encoded data to a server in an HTTP request;

[1263] 10. The system of claim 1, further comprising:

[1264] (Claim 3)

[1265] A means for the server to decode the received video data and analyze it using an AI model;

[1266] A method to encode the analysis results in JSON format and send them to the terminal via an HTTP request,

[1267] 10. The system of claim 1, further comprising:

[1268] "Example 1"

[1269] (Claim 1)

[1270] a means for acquiring video data;

[1271] means for pre-processing and encoding the acquired video data;

[1272] means for transmitting the preprocessed and encoded data to a server;

[1273] A means for the server to decode the received video data and analyze it using an AI model;

[1274] A means to encode the analysis results in JSON format and send them to the terminal as a judgment result;

[1275] means for notifying a user of the determination result received by the terminal;

[1276] A system including:

[1277] (Claim 2)

[1278] a means for capturing video data in real time;

[1279] A means for pre-processing the captured video data to remove noise and adjust the frame rate;

[1280] means for encoding the pre-processed video data into a suitable format;

[1281] a means for transmitting the encoded data to a server in an HTTP request;

[1282] 10. The system of claim 1, further comprising:

[1283] (Claim 3)

[1284] The server encodes the judgment results generated based on the output of the AI ​​model in JSON format and sends them to the terminal via an HTTP request.

[1285] means for notifying the user of the determination result received by the terminal by displaying it on a screen and by audio notification;

[1286] 10. The system of claim 1, further comprising:

[1287] "Application Example 1"

[1288] (Claim 1)

[1289] a means for acquiring video data;

[1290] means for transmitting the acquired video data to a server;

[1291] A means for the server to analyze the video data and generate a determination result;

[1292] means for transmitting the generated determination result to a terminal;

[1293] means for notifying a user of the determination result received by the terminal;

[1294] Using generative AI models to detect suspicious behavior and people in real time;

[1295] a means for executing an AI model based on a prompt;

[1296] A system including:

[1297] (Claim 2)

[1298] a means for capturing video data in real time;

[1299] means for encoding the captured video data into an appropriate format;

[1300] a means for transmitting the encoded data to a server in an HTTP request;

[1301] 10. The system of claim 1, further comprising:

[1302] (Claim 3)

[1303] A means for the server to decode the received video data and analyze it using an AI model;

[1304] A method to encode the analysis results in JSON format and send them to the terminal via an HTTP request,

[1305] A means for generating a list of suspicious behaviors or people as a result of the analysis and notifying the terminal of the list based on a prompt sentence;

[1306] 10. The system of claim 1, further comprising:

[1307] "Example 2: Combining Emotion Engines"

[1308] (Claim 1)

[1309] a means for acquiring video data;

[1310] means for transmitting the acquired video data to a server;

[1311] A means for the server to analyze the video data and generate a determination result;

[1312] means for transmitting the generated determination result to a terminal;

[1313] means for notifying a user of the determination result received by the terminal;

[1314] A means for recognizing a user's emotion and adjusting the notification content;

[1315] A system including:

[1316] (Claim 2)

[1317] a means for capturing video data in real time;

[1318] means for encoding the captured video data into an appropriate format;

[1319] a means for transmitting the encoded data to a server in an HTTP request;

[1320] a means for capturing a user's facial expressions and voice;

[1321] 10. The system of claim 1, further comprising:

[1322] (Claim 3)

[1323] A means for the server to decode the received video data and analyze it using a generative AI model;

[1324] A method to encode the analysis results in JSON format and send them to the terminal via an HTTP request,

[1325] A means for analyzing user emotions and dynamically adjusting notification content based on the analysis results;

[1326] 10. The system of claim 1, further comprising:

[1327] "Application example 2 when combining emotion engines"

[1328] (Claim 1)

[1329] a means for acquiring video data;

[1330] means for transmitting the acquired video data to a server;

[1331] A means for the server to analyze the video data and generate a determination result;

[1332] means for transmitting the generated determination result to a terminal;

[1333] means for notifying a user of the determination result received by the terminal;

[1334] means for dynamically adjusting notification content using an emotion engine that recognizes user emotions;

[1335] A system including:

[1336] (Claim 2)

[1337] a means for capturing video data in real time;

[1338] means for encoding the captured video data into an appropriate format;

[1339] a means for transmitting the encoded data to a server in an HTTP request;

[1340] a means for performing quality control in a factory environment using an artificial intelligence model;

[1341] 10. The system of claim 1, further comprising:

[1342] (Claim 3)

[1343] A means for the server to decode the received video data and analyze it using a generative AI model;

[1344] A method to encode the analysis results in JSON format and send them to the terminal via an HTTP request,

[1345] means for analyzing the video data based on the prompt sentence to determine the worker's emotion;

[1346] 10. The system of claim 1, further comprising: [Explanation of symbols]

[1347] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring video data; means for transmitting the acquired video data to a server; A means for the server to analyze the video data and generate a determination result; means for transmitting the generated determination result to a terminal; means for notifying a user of the determination result received by the terminal; A system including:

2. a means for capturing video data in real time; means for encoding the captured video data into an appropriate format; a means for transmitting the encoded data to a server in an HTTP request; The system of claim 1 further comprising:

3. A means for the server to decode the received video data and analyze it using an AI model; A method to encode the analysis results in JSON format and send them to the terminal via an HTTP request, The system of claim 1 further comprising:

Images