system

Abstract

A system is provided.SOLUTION: A system, comprising: means for identifying a person in an image or video of a news article using image recognition technology; means for searching for an official social networking service (SNS) account of the identified person; and means for enabling a user to follow the official SNS account in a news application.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People who want to know the official social networking service (SNS) accounts of people they see in news articles have had to search for the person's name, check it in a browser, and then follow the official account, which was a time-consuming process. Furthermore, there have been problems with not knowing which SNS accounts the person owns, and with the existence of fake accounts. This invention solves these problems by making it easy to follow the official SNS accounts of people who appear in news articles. [Means for solving the problem]

[0005] This invention provides a system that uses image recognition technology to identify people appearing in images and videos of news articles. Specifically, the system extracts features of people from images and videos within a news application and identifies them by comparing them with a database based on those features. It also searches for the official social media accounts of identified people and provides a means for users to follow the official accounts within the news application, significantly reducing the user's effort and mitigating the problem of fake accounts.

[0006] "Image recognition technology" is a technology that automatically detects and interprets specific information, features, or patterns from images or videos.

[0007] A "news article" is a collection of news or commentary text, images, and video provided online or in print.

[0008] A "person" refers to a specific individual who appears in an image or video, and is an object that can be identified based on the individual's features.

[0009] A "social networking service (SNS) account" is a profile created by an individual or organization on an SNS platform, along with any associated posts, followers, and other accounts that the individual or organization follows.

[0010] A "news application" is a software application that a user uses to view news articles.

[0011] "Features" are information extracted from images or videos, such as facial shape or biological features, which are quantified data used to identify a person.

[0012] A "database" is a collection of data designed to centrally manage large amounts of data and make it easy to search and compare.

[0013] An "official account mark" is a specific symbol or verification badge given to an account on a social media platform to indicate that it is official. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention provides a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles and allows users to easily follow those accounts. The program for this system is described in detail below.

[0036] Overall system overview

[0037] The system consists of three main components: a server, a terminal, and a user.

[0038] 1. Server

[0039] The server receives image and video data from the news application, identifies people using image recognition technology, searches for the identified people's official social media accounts on various social media platforms, and sends that information to the device.

[0040] 2. Terminal

[0041] The terminal is a device on which a news application operated by a user runs. The terminal displays the official SNS account information received from the server on a user interface, and when the user presses the follow button, the terminal sends the request to the server.

[0042] 3. Users

[0043] Users operate the news application to view news articles they are interested in. They can easily follow the official social media accounts of people featured in images and videos within the articles.

[0044] Program processing

[0045] 1. Image and video analysis

[0046] The device acquires image and video data from the news article the user is viewing and sends it to a server. The server then uses image recognition technology to identify people appearing in the images and videos based on the received data. The server then uses the latest AI models to extract features and compare them with a database to identify the person.

[0047] 2. Search for official social media accounts

[0048] After the person is identified, the server searches for the identified person's social media accounts on various social media platforms. Specifically, the server collects the official social media accounts via API based on the identified person's name and stores them in a database.

[0049] 3. Provide the user with account information

[0050] The server sends the search results to the device, which then displays the official social media account information within the news application, allowing users to view the official accounts of their interest on the news application interface.

[0051] 4. Implementing the Follow Feature

[0052] When a user presses the follow button from the displayed list of official accounts, the device sends the request to the server. The server automatically follows the official account based on the user's SNS authentication information. The server notifies the device of the completion of the follow and displays it to the user.

[0053] Specific examples

[0054] Scenario: A user views a news article featuring a famous actor.

[0055] 1. A user opens an article in a news application that features a famous actor.

[0056] 2. The device retrieves the image and video data from the article and sends it to the server.

[0057] 3. The server analyzes the received data using an "AI model" and extracts the actor's characteristics.

[0058] 4. The server compares the extracted features with a database to identify famous actors.

[0059] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[0060] 6. The device displays the received official account information in the news application.

[0061] 7. The user presses the follow button from the displayed list of official accounts.

[0062] 8. The device sends a request to the server that the follow button was pressed.

[0063] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[0064] 10. The server notifies the terminal of the results of the follow process, and the terminal displays a message to the user indicating that the follow has been completed.

[0065] This will create a system that allows users to easily follow the official social media accounts of people featured in news articles.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] A user navigates through a news application and opens a news article of interest.

[0069] Step 2:

[0070] The device extracts image and video data contained in the news article and sends that data to the server.

[0071] Step 3:

[0072] The server inputs the received image and video data into the AI ​​model and extracts the features of the people appearing in the images and videos.

[0073] Step 4:

[0074] The server compares the extracted features with an internal database to identify the person being displayed.

[0075] Step 5:

[0076] Based on the information of the identified person, the server uses the APIs of various social media platforms to search for that person's official social media accounts.

[0077] Step 6:

[0078] The server identifies accounts that have official account marks (e.g., verification badges) and extracts this as official account information.

[0079] Step 7:

[0080] The server sends official SNS account information to the device.

[0081] Step 8:

[0082] The device displays the received official SNS account information on the interface within the news application.

[0083] Step 9:

[0084] The user clicks the follow button from the displayed list of official accounts.

[0085] Step 10:

[0086] When the user clicks the follow button, the device sends a follow request to the server, including the user's SNS authentication information.

[0087] Step 11:

[0088] The server receives the follow request and performs authentication procedures on each SNS platform based on the user's SNS authentication information.

[0089] Step 12:

[0090] After the server successfully authenticates the user, it uses the API to follow the target official account from the user's account.

[0091] Step 13:

[0092] The server notifies the terminal of the result of the follow-up process.

[0093] Step 14:

[0094] The device notifies the user by displaying a message in the news application indicating that the follow has been completed.

[0095] This series of processes allows users to easily follow the official social media accounts of people featured in news articles.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Conventional news applications lack a means to quickly and accurately identify the official social networking service (SNS) accounts of people featured in articles and allow users to easily follow them. Furthermore, analyzing images and videos takes time, making it difficult to provide accurate account information.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes a means for a terminal to acquire and transmit image and video data of a news article, a means for the server to analyze the received image and video data and identify a person using image recognition technology, a means for the server to search for the identified person's official social networking service (SNS) account from various SNS platforms, a means for the server to perform a follow process based on the user's SNS authentication information, and a means for the terminal to display official SNS account information to the user and provide a follow button, thereby enabling the user to quickly and accurately identify the official SNS account of a person appearing in a news article and easily follow the person.

[0101] A "terminal" is a device on which a user operates a news application, and is generally an electronic device such as a smartphone, tablet, or PC.

[0102] "Images and video data of news articles" refers to visual content contained in articles provided within the news application, including still images and video data.

[0103] "Server" refers to a central processing unit that processes data received from a terminal and transmits analysis results and additional information to the terminal.

[0104] "Image recognition technology" refers to technology that uses computer vision techniques to identify people and objects in images and videos.

[0105] An "SNS platform" is an online system that provides social networking services, and generally refers to networks such as Twitter and Facebook.

[0106] An "AI model" refers to mathematical algorithms and data structures created based on artificial intelligence technology to automatically perform specific tasks (such as person identification).

[0107] "Features" refer to important attributes and patterns extracted from images and videos, and are used to identify people and objects.

[0108] "Official social media account" refers to an official social media account that is verified on a social media platform, such as the official profile of a celebrity or company.

[0109] A "follow button" is a button that is displayed on the user interface, and a user can click on it to follow a specific SNS account.

[0110] MODE FOR CARRYING OUT THE INVENTION

[0111] This invention provides a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles and enables users to easily follow those accounts. Detailed embodiments of this system are described below.

[0112] System Configuration

[0113] This system mainly consists of a server, terminals, and users.

[0114] 1. Server

[0115] The server receives image and video data sent from the news application and uses image recognition technology to identify people.

[0116] The server searches for the identified person's official social media account on various social media platforms and sends that information to the device.

[0117] The server is equipped with AI models such as TensorFlow and OpenCV, which are used to analyze the received data.

[0118] 2. Terminal

[0119] The terminal is a device on which a news application operated by a user runs, and includes a smartphone, tablet, PC, etc.

[0120] The terminal displays the official SNS account information received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[0121] 3. Users

[0122] The user operates the news application to view news articles of interest.

[0123] Users can easily follow the official social media accounts of people featured in images and videos within articles using the follow button.

[0124] Operation overview

[0125] Data Acquisition and Transmission

[0126] When a user uses a news application to view a news article, the device acquires the image and video data contained in the article and transmits it to a server.

[0127] Analysis and Identification

[0128] The server uses the latest AI models to extract features from the received image and video data, and compares them with a database to identify people, using image recognition technologies such as TensorFlow and OpenCV.

[0129] Account Search

[0130] The server searches for official accounts using the APIs of various social media platforms based on the identified person's name. The collected social media account information is temporarily stored in a database.

[0131] Information provision

[0132] The server sends the search results to the terminal, and the terminal displays the official SNS account information on the user interface.

[0133] Follow-up process

[0134] When a user presses the follow button, the device sends the request to the server. The server automatically follows the official account based on the user's SNS authentication information. The device is notified of the success or failure of this process, and displays a message to the user indicating that the follow has been completed.

[0135] Specific examples

[0136] Scenario: A user views a news article featuring a famous actor.

[0137] 1. A user opens an article in a news application that mentions a famous actor.

[0138] 2. The device retrieves the image and video data from the article and sends it to the server.

[0139] 3. The server analyzes the received data using TensorFlow and extracts the actor's features.

[0140] 4. The server compares the extracted features with a database to identify famous actors.

[0141] 5. The server searches for the official social media account using the identified actor's name using the "Facebook API" or "Twitter API" and sends the official account information to the device.

[0142] 6. The official account information received by the device is displayed within the news application.

[0143] 7. The user clicks the Follow button from the displayed list of official accounts.

[0144] 8. The device sends a request to the server that the follow button has been pressed.

[0145] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[0146] 10. The server notifies the device of the results of the follow process, and the device displays a message to the user indicating that the follow has been completed.

[0147] This system allows users to quickly and accurately identify the official social media accounts of people featured in news articles and easily follow them.

[0148] Example prompts to input to the generative AI model

[0149] prompt:

[0150] Describe a system that identifies the official social media accounts of celebrities mentioned in news articles. This system analyzes image and video data from news articles and provides users with the social media account information of the identified people, allowing them to easily follow them.

[0151] scenario:

[0152] 1. A user opens an article in a news application that features a famous actor.

[0153] 2. The device retrieves the image and video data from the article and sends it to the server.

[0154] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[0155] 4. The server compares the extracted features with a database to identify famous actors.

[0156] 5. The server uses an API to search for the official social media account for the identified actor's name and sends the official account information to the device.

[0157] 6. The device displays the received official account information in the news application.

[0158] 7. The user presses the follow button from the displayed list of official accounts.

[0159] 8. The device sends a request to the server that the follow button was pressed.

[0160] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[0161] 10. The server notifies the terminal of the results of the follow process, and the terminal displays a message to the user indicating that the follow has been completed.

[0162] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0163] Program processing flow

[0164] Step 1:

[0165] A user launches a news application and browses to a news article of interest.

[0166] Specifically, a user selects an article featuring a famous actor from the application's news feed, opens the article, and begins reading.

[0167] Input: User launches news application and selects article

[0168] Output: Display article content

[0169] Step 2:

[0170] The device acquires image and video data from the news article the user is viewing.

[0171] The device automatically extracts all image and video data contained in the article and temporarily stores it.

[0172] Input: News article data

[0173] Output: Extraction of image and video data

[0174] Step 3:

[0175] The image and video data acquired by the device is sent to the server.

[0176] The terminal collects all the extracted image and video data into packets and transmits them to the server.

[0177] Input: Image and video data

[0178] Output: Send data to the server

[0179] Step 4:

[0180] The image and video data received by the server is analyzed using an AI model (e.g., TensorFlow).

[0181] The server inputs the received data into an AI model and extracts the person's features, using facial recognition technology.

[0182] Input: Image and video data

[0183] Output: Feature extraction

[0184] Step 5:

[0185] The server identifies the person based on the extracted features by comparing them with a database.

[0186] The server compares the extracted features with a database of pre-trained people and identifies matching people.

[0187] Input: Feature data, database

[0188] Output: Identified person

[0189] Step 6:

[0190] The server searches for the official social media accounts of the identified person from various social media platforms (e.g., Facebook API, Twitter API).

[0191] The server uses the API of the social media platform to search for official accounts based on the identified person's name and collects that information.

[0192] Input: Name of the identified person

[0193] Output: Official social media account information

[0194] Step 7:

[0195] The server sends the collected official SNS account information to the device.

[0196] The server assembles the collected account information into packets and sends them to the terminal.

[0197] Input: Official SNS account information

[0198] Output: Send account information to the device

[0199] Step 8:

[0200] The SNS account information received by the device is displayed within the news application.

[0201] The device displays the received official SNS accounts in a list format on the user interface.

[0202] Input: Official SNS account information

[0203] Output: Display account information

[0204] Step 9:

[0205] The user clicks the follow button for the account of interest from the displayed list of official accounts.

[0206] The user selects the desired account from the displayed list and clicks the follow button.

[0207] Input: Official SNS account list

[0208] Output: Follow button click

[0209] Step 10:

[0210] The device sends a request to the server that the follow button has been pressed.

[0211] The terminal sends the user's follow request to the server, and the request includes the user's SNS authentication information.

[0212] Input: Follow request

[0213] Output: Request sent to server

[0214] Step 11:

[0215] The server follows the official account based on the user's social media authentication information.

[0216] The server executes the follow process using the SNS API based on the received request.

[0217] Input: Follow request, social media credentials

[0218] Output: Follow processing result

[0219] Step 12:

[0220] The server notifies the terminal of the follow-up processing result.

[0221] The server notifies the terminal of the success or failure of the follow process.

[0222] Input: Follow-up processing result

[0223] Output: Notification to terminal

[0224] Step 13:

[0225] The device displays a follow-up completion message to the user.

[0226] The terminal displays a message to the user indicating that the follow has been completed based on the result of the follow process received from the server.

[0227] Input: Notification from the server

[0228] Output: Display of follow-up completion message

[0229] (Application example 1)

[0230] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0231] There is a need for a system that can identify the official social media accounts of people appearing in images and videos displayed on web pages and allow users to easily follow them, but with existing technology it has been difficult to automate this process, and users have had to perform cumbersome operations. There is a need to solve these issues and improve the user experience.

[0232] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0233] In this invention, the server includes means for identifying a person appearing in an image or video on a webpage using image recognition technology, means for searching for the official information acquisition service account of the identified person, means for allowing a user to follow the official information acquisition service account within the website, means for displaying the official information acquisition service account information of the identified person on a user interface, and means for automatically following the official information acquisition service account based on the account information acquired by the user. This allows a user to easily follow a person's official SNS account without performing complicated operations.

[0234] "Image recognition technology" is a technology that extracts useful information from video data and identifies people and objects.

[0235] A "web page" is a document containing information that is viewable on the Internet.

[0236] "Images and videos" refers to still images and moving video data that contain visual information.

[0237] "People" refers to people appearing in an image or video.

[0238] An "official information acquisition service account" is an account that is authenticated on a social networking service (SNS).

[0239] "Searching" is the process of locating specific information.

[0240] A "user" is a person who uses the system.

[0241] A "user interface" is a screen and operating means for a user to interact with a system.

[0242] "Following" refers to linking with a specific account on social media and receiving information.

[0243] A "server" is a computer system that processes data on a network.

[0244] The system for implementing this invention consists of three main components: a server, a terminal, and a user.

[0245] 1. Server Processing

[0246] The server receives image and video data from web pages and uses image recognition technology to identify people. Software such as TensorFlow and PIL (Python Imaging Library) is installed on the server, which processes and calculates the received image and video data. Specifically, features are extracted from the image data and a generative AI model is used to match the data with a database to identify the person. The identified person's official social media account is searched for via the API of each social media platform. The search results are saved for subsequent processing.

[0247] 2. Terminal Processing

[0248] The terminal is a device that displays the web page operated by the user, and has the role of displaying the account information of the official information acquisition service received from the server on the user interface. When the user presses the follow button, the request is sent from the terminal to the server.

[0249] 3. User Operation

[0250] While browsing a website, users may want to follow the official information service account of a person who appears in an image or video on the page. In this case, users can easily follow the official account by pressing the follow button displayed on their device.

[0251] Specific examples

[0252] For example, suppose a user is browsing a favorite product page on a fashion shopping site. The product page features an image of a famous model. Using this system, the following happens:

[0253] 1. When a user opens a product page, the device sends the image data of that page to the server.

[0254] 2. The server uses TensorFlow to perform image analysis and extract model features.

[0255] 3. Based on the features, the server matches the database and identifies the model.

[0256] 4. The server searches for the identified model's official information retrieval service account (e.g., Instagram or Twitter) using the API of each social media platform.

[0257] 5. The retrieved account information is sent to the device and displayed on the user's web page.

[0258] 6. Users can follow the model's official account from their own social media account by clicking the follow button that appears.

[0259] Prompt Sentence Examples

[0260] Example prompts to input to generative AI models such as Chat GPT:

[0261] We are developing a system that uses image analysis technology to identify the official social media accounts of people featured in specific news articles and automatically follow those accounts. Please generate a prompt sentence under the following conditions:

[0262] The system first retrieves an image / video of the page the user is viewing.

[0263] The captured images / videos are sent to the server and people are identified using TensorFlow.

[0264] Search for the official social media accounts of the identified person using the social media API.

[0265] Displays SNS account information to users. Users can follow official accounts by clicking the follow button.

[0266] Finally, we display a message to the user confirming that they have followed us.

[0267] This allows a system to be realized that allows users to easily follow a person's official SNS account when browsing a web page.

[0268] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0269] Step 1:

[0270] The device acquires image and video data from the web page the user is viewing. The URL of the web page and a reference image are given as input. The device then sends the acquired image data to the server. In this step, the device collects image data using a browser or a specific application installed on the device, converts the received data into an appropriate format, and passes it to the server.

[0271] Step 2:

[0272] The server processes the image and video data received from the device. First, it uses image recognition technology to extract the features of the people in the image. Image data is given as input, and the person's identification information is obtained as output. Then, TensorFlow is used to perform image analysis and extract specific features (such as facial contours and the placement of the eyes, nose, and mouth). This generates data that can be used to identify specific people in the image or video.

[0273] Step 3:

[0274] The server uses the extracted features to match the database and identify the person. Feature data is given as input, and the identified person's identity is obtained as output. The server uses a generative AI model to quickly search the existing database and return the corresponding person information. This step identifies who the person in the image is.

[0275] Step 4:

[0276] The server searches each SNS platform for the identified person's official information retrieval service account. The person's identification information is given as input, and SNS account information is obtained as output. The server uses the SNS API to send a search query based on the identified person's name and collects official account information. In this step, only authenticated accounts are extracted and listed.

[0277] Step 5:

[0278] The server sends the identified official information retrieval service account information to the device. The social media account information is given as input, and account information displayed in the user interface is generated as output. The server sends formatted data so that the information can be displayed correctly on the device. This data includes the official mark, account name, and profile URL.

[0279] Step 6:

[0280] The device displays the received official information acquisition service account information on a user interface. The account information sent from the server is given as input, and information that can be visually confirmed by the user is displayed as output. The device appropriately arranges the information on a web page and provides interactive elements including a follow button.

[0281] Step 7:

[0282] The user presses the Follow button from the displayed list of official information acquisition service accounts. The user's action is taken as input, and a follow request is generated as output. In this step, the Follow button responds to the user's action as a trigger, and the next action is taken.

[0283] Step 8:

[0284] When the user presses the follow button, the device sends a follow request to the server. User operation information is given as input, and the follow request is sent to the server as output. The device generates follow request data and makes a specific API call to the server.

[0285] Step 9:

[0286] The server receives a follow request from a user and automatically follows a specific official information retrieval service account based on that information. The follow request information is given as input, and a follow completion status is generated as output. The server executes the follow action through the SNS API.

[0287] Step 10:

[0288] The server notifies the device of the results of the follow process, and the device displays a message to the user indicating that the follow has been completed. The follow completion information is given as input, and a confirmation message is generated as output and displayed to the user. The server and device work together to allow the user to confirm that the follow was successful.

[0289] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0290] This invention combines a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles, allowing users to easily follow those accounts, with an emotion engine that recognizes the user's emotions. The program for this system is described in detail below.

[0291] Overall system overview

[0292] The system consists of three main components: a server, a terminal, and a user.

[0293] 1. Server

[0294] The server receives image and video data from the news application and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It also uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow.

[0295] 2. Terminal

[0296] The terminal is a device that runs a news application operated by the user. The terminal displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[0297] 3. Users

[0298] Users operate the news application to browse news articles of interest. They can easily follow the official social media accounts of people featured in images and videos within the article. They can also select more appropriate follow-up actions by receiving suggestions based on the analysis results of the emotion engine.

[0299] Program processing

[0300] 1. Image and video analysis

[0301] When a user operates a news application and opens a news article, the device retrieves the images and video data from the article and sends them to the server. The server then inputs the received image and video data into an AI model and extracts the features of the people appearing in the images and videos.

[0302] 2. Identifying people and searching social media accounts

[0303] The server identifies the person displayed by checking the extracted features against an internal database. It then searches for the identified person's official social media accounts via the APIs of various social media platforms, identifies accounts with the official account mark (verification badge), and extracts this as official account information.

[0304] 3. User Emotion Recognition and Analysis

[0305] The device captures the user's facial expressions, voice, operation patterns, etc. in real time and sends this data to the server. The server uses an emotion engine to analyze this data and recognize the user's emotional state. Emotional information indicates the user's basic emotional state, such as joy, anger, sadness, and happiness.

[0306] 4. Providing account information and sentiment analysis results

[0307] The server creates suggestions for following SNS accounts that match the user's emotions based on the official SNS account information and the results of the user's emotion analysis, and sends the suggestions to the device. The device then displays the received information on the interface within the news application.

[0308] 5. Implementing the Follow Feature

[0309] The user presses the follow button from the list of official accounts displayed and suggestions based on the sentiment analysis results. When the follow button is pressed, the device sends a follow request to the server. This request also includes the user's SNS authentication information.

[0310] 6. Execute follow-up process

[0311] The server receives the follow request and performs authentication procedures on each SNS platform based on the SNS authentication information. After successful authentication, the API is used to follow the target official account from the user's account. The server notifies the device of the follow processing results, and the device displays a message to the user indicating that the follow has been completed.

[0312] Specific examples

[0313] Scenario: A user views a news article featuring a famous actor.

[0314] 1. A user opens an article in a news application that mentions a famous actor.

[0315] 2. The device retrieves the image and video data from the article and sends it to the server.

[0316] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[0317] 4. The server compares the features with the database to identify famous actors.

[0318] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[0319] 6. The device acquires the user's facial expressions, voice, and operation patterns and sends them to the server.

[0320] 7. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the device.

[0321] 8. The device displays the received information within the app.

[0322] 9. The user presses the Follow button from the displayed list of accounts and suggestions.

[0323] 10. The device sends the follow button click information to the server.

[0324] 11. The server performs authentication procedures on each SNS based on the SNS authentication information and completes the follow process using the API.

[0325] 12. The server sends the results of the follow process to the terminal, and the terminal displays a message to the user indicating that the follow has been completed.

[0326] This system allows users to easily follow the official social media accounts of people featured in news articles based on sentiment-based suggestions.

[0327] The processing flow will be explained below.

[0328] Step 1:

[0329] A user navigates through a news application and opens a news article of interest.

[0330] Step 2:

[0331] The device acquires image and video data from the article and sends that data to the server.

[0332] Step 3:

[0333] The server inputs the received image and video data into the AI ​​model and extracts the features of the people appearing in the images and videos.

[0334] Step 4:

[0335] The server compares the extracted features with an internal database to identify the person being displayed.

[0336] Step 5:

[0337] Based on the information of the identified person, the server uses the APIs of various social media platforms to search for that person's official social media accounts.

[0338] Step 6:

[0339] The server identifies accounts that have official account marks (e.g., verification badges) and extracts this as official account information.

[0340] Step 7:

[0341] The terminal acquires data such as the user's facial expressions, voice, and operation patterns, and sends this data to the server.

[0342] Step 8:

[0343] The server inputs the received data into the emotion engine and analyzes the user's emotional state. For example, if the user smiles, it identifies the emotional state as "joy," and if the user frowns, it identifies the emotional state as "dissatisfaction."

[0344] Step 9:

[0345] Based on the results of the emotion analysis, the server creates suggestions for following official social media accounts that match the user's emotions.

[0346] Step 10:

[0347] The server sends official SNS account information and emotion analysis results to the device.

[0348] Step 11:

[0349] The device will display the received official social media account information and follow suggestions on the interface within the news application.

[0350] Step 12:

[0351] Users can click the follow button from the displayed list of official accounts and suggestions based on sentiment analysis results.

[0352] Step 13:

[0353] When the user clicks the follow button, the device sends a follow request to the server, including the user's SNS authentication information.

[0354] Step 14:

[0355] The server receives the follow request and performs authentication procedures on each SNS platform based on the user's SNS authentication information.

[0356] Step 15:

[0357] After successful authentication, the server uses the API to follow the target official account from the user's account.

[0358] Step 16:

[0359] The server notifies the terminal of the result of the follow-up process.

[0360] Step 17:

[0361] The device notifies the user by displaying a message in the news application indicating that the follow has been completed.

[0362] Example 2

[0363] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] In conventional news applications, when users want to follow the official social networking service (SNS) accounts of people featured in news articles, they must manually search for them, which is time-consuming. Furthermore, there is insufficient information to determine whether the account is of interest to the user, and simply providing account information does not sufficiently stimulate the user's interest. This not only requires time and effort to follow, but also results in a lack of improvement in the user experience.

[0365] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0366] In this invention, the server includes means for identifying people appearing in images and videos of news articles using image recognition technology, means for searching for the identified people's official social networking service (SNS) accounts, means for allowing users to follow the official SNS accounts within the news application, means for acquiring the user's emotional data in real time, means for analyzing the user's emotional state using an emotion engine, and means for creating follow suggestions based on the analyzed emotional information. This allows users to easily follow the official SNS accounts of people appearing in news articles, and further provides a more engaging experience by receiving appropriate follow suggestions based on emotions.

[0367] "Image recognition technology" is a technology that identifies and analyzes specific features and patterns from digital images and videos.

[0368] A "news article" refers to the informational content published in the media, including multimedia content such as text, images, and video.

[0369] "Person identification" is the process of identifying people in images or videos using specific methods and algorithms.

[0370] A "social networking service (SNS)" is an online platform that enables users to communicate with each other over the Internet.

[0371] An "official SNS account" is an account officially authenticated by an individual or organization on a social networking service.

[0372] "Following" is the act of registering another user on a social networking site with the aim of receiving their posts and information on a regular basis.

[0373] "Emotion data" is data that indicates the user's emotional state, and is obtained from facial expressions, voice, operation patterns, and the like.

[0374] An "emotion engine" is software or an algorithm that analyzes a user's emotional state based on collected emotional data.

[0375] "Follow suggestions" are suggestions that recommend social media accounts for users to follow based on analyzed information and the user's interests.

[0376] A "server" is a computer system that processes data and provides information in response to user requests.

[0377] A "terminal" is a device that allows a user to operate a news application, and includes a smartphone, tablet, or the like.

[0378] This invention is a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles, allowing users to easily follow those accounts, and combines this with an emotion engine that recognizes the user's emotions. This system consists of three main components: a server, a terminal, and a user.

[0379] Overall system overview

[0380] 1. Server

[0381] The server receives image and video data from the news application and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It then uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow.

[0382] 2. Terminal

[0383] The terminal is a device that runs a news application operated by the user. The terminal displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[0384] 3. Users

[0385] Users operate the news application to browse news articles of interest. They can easily follow the official social media accounts of people featured in images and videos within the article. They can also select more appropriate follow-up actions by receiving suggestions based on the analysis results of the emotion engine.

[0386] Hardware and software used

[0387] The server is a server machine equipped with a high-performance processor that runs an AI model using Python (e.g., TensorFlow, OpenCV). The emotion engine uses an emotion analysis service such as Microsoft Azure Emotion API. The terminal is a device such as a smartphone or tablet that exchanges data with the server in real time using WebSocket technology. The terminal is also equipped with a camera and microphone and has the ability to capture the user's facial expressions and voice.

[0388] Specific examples

[0389] Scenario: A user views a news article featuring a famous actor.

[0390] 1. A user opens an article in a news application that mentions a famous actor.

[0391] 2. The device retrieves the image and video data from the article and sends it to the server.

[0392] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[0393] 4. The server compares the features with the database to identify famous actors.

[0394] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[0395] 6. The device acquires the user's facial expressions, voice, and operation patterns and sends them to the server.

[0396] 7. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the device.

[0397] 8. The device displays the received information within the app.

[0398] 9. The user presses the Follow button from the displayed list of accounts and suggestions.

[0399] 10. The device sends the follow button click information to the server.

[0400] 11. The server performs authentication procedures on each SNS based on the SNS authentication information and completes the follow process using the API.

[0401] 12. The server sends the results of the follow process to the terminal, and the terminal displays a message to the user indicating that the follow has been completed.

[0402] Prompt Sentence Examples

[0403] "Please explain how you input image data into your AI model and extract features of people in the image. Please be specific about the process, including the specific libraries and tools you use."

[0404] This system allows users to easily follow the official social media accounts of people featured in news articles based on sentiment-based suggestions.

[0405] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0406] Step 1:

[0407] A user launches a news application and selects a particular news article, which includes images and video.

[0408] Input: User action to select a news article

[0409] Output: Start retrieving image and video data from articles

[0410] What happens: A user opens a news application on their smartphone and clicks on an article of interest from the news feed.

[0411] Step 2:

[0412] The device acquires image and video data from the article, and sends the acquired data to the server.

[0413] Input: URL of image or video in news article

[0414] Output: Binary image and video data

[0415] Specific operation: The device extracts the URLs of images and videos from the DOM of the article displayed by the news application, and sends the obtained image and video data in binary format to the server.

[0416] Step 3:

[0417] The server inputs the received image and video data into the AI ​​model and extracts the features of the people in the images.

[0418] Input: binary image or video data

[0419] Output: Extracted person feature data

[0420] How it works: The server uses Python libraries (e.g., TensorFlow, OpenCV) to input image data into the AI ​​model, and the model extracts facial features (eye position, nose shape, jaw line, etc.).

[0421] Step 4:

[0422] The server compares the extracted features with an internal database to identify the person being displayed.

[0423] Input: Extracted person feature data

[0424] Output: Data of identified person

[0425] Specific operation: The server compares the extracted feature data with an internal database and obtains the ID and name of the identified person.

[0426] Step 5:

[0427] The server searches for the identified person's official social media accounts using the APIs of various social media platforms, extracts official accounts (with verification badges) from the search results, and sends that information to the device.

[0428] Input: ID or name of the identified person

[0429] Output: Official social media account information (with verification badge)

[0430] Specific operation: The server uses the Twitter API or Facebook Graph API to search for official social media accounts using the identified person's name, extracts official account information with a verification badge from the obtained account data, and sends it to the device in JSON format.

[0431] Step 6:

[0432] The device acquires emotional data such as the user's facial expressions, voice, and operation patterns in real time and transmits this data to the server.

[0433] Input: User's facial expression, voice, and operation pattern data

[0434] Output: Collected emotion data

[0435] Specific operation: The device uses a camera and microphone to capture the user's facial expressions and voice, and also records operation data such as the user's touch pattern and scrolling speed, and sends it to the server via WebSocket.

[0436] Step 7:

[0437] The server uses an emotion engine to analyze the received emotion data and recognize the user's emotional state. Based on the results, it creates suggestions for following SNS accounts suitable for the user and sends them to the device.

[0438] Input: Collected emotion data

[0439] Output: Sentiment analysis results and follow suggestion data

[0440] How it works: The server uses an emotion analysis library (e.g., Microsoft Azure Emotion API) to analyze facial expressions and voice data to identify the user's emotional state (joy, anger, sadness, etc.), and then generates follow suggestions based on this and sends the information to the device.

[0441] Step 8:

[0442] The device displays the received SNS account information and emotion analysis results on the user interface and sends a request to the server for the user to press the follow button.

[0443] Input: Follow suggestion data

[0444] Output: User's follow button operation information

[0445] Specific operation: The device displays official social media account information and sentiment analysis results on the interface within the news app, receives instructions from the user to press the follow button for accounts that interest them, and sends this information to the server.

[0446] Step 9:

[0447] The server receives the click information of the follow button, performs authentication procedures with the corresponding SNS based on the SNS authentication information, and then completes the follow process using the API.

[0448] Input: User follow button operation information, SNS authentication information

[0449] Output: Follow processing result

[0450] Specific operation: The server uses the SNS API to send a follow request based on the user's SNS authentication information and receives a response indicating whether the follow process was successful.

[0451] Step 10:

[0452] The server notifies the terminal of the result of the follow process, and the terminal displays a message indicating that the follow has been completed to the user.

[0453] Input: Follow-up processing result data

[0454] Output: Follow completion message

[0455] Specific operation: The server sends data to the device indicating that the follow process has been completed, and the device then pops up a "Follow Complete" message to the user based on that data.

[0456] This series of processes allows users to easily follow the official social media accounts of people featured in news articles, and by receiving appropriate follow suggestions based on their emotions, users can have a more engaging experience.

[0457] (Application example 2)

[0458] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0459] Conventional news applications have the drawback of requiring users to identify and follow the social media accounts of people featured in articles, and of not suggesting appropriate accounts to follow that reflect the user's interests. Furthermore, in the advertising field, there is a lack of functionality that allows users to easily identify the social media accounts of people featured in advertisements and follow them based on their emotions. Therefore, there is a need for a method to increase user engagement and maximize advertising effectiveness.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0461] In this invention, the server includes means for identifying people appearing in images and videos of news articles using image recognition technology, means for searching for the official social networking service (SNS) accounts of the identified people, means for analyzing the user's emotions using an emotion engine that recognizes the user's emotions, means for suggesting appropriate official SNS accounts to follow based on the analyzed emotions, and means for identifying people in advertisements based on image and video analysis of the advertisements and searching for their SNS accounts. This enables users to easily follow the official SNS accounts of people appearing in news articles and advertisements based on emotion-based suggestions.

[0462] "Image recognition technology" is a technology for identifying and analyzing people and objects from images and video data acquired using cameras and sensors.

[0463] A "news article" is a document or content that describes news reports published in newspapers, websites, etc.

[0464] A "video" is a medium that expresses movement by playing back a series of multiple images along a time axis.

[0465] "Person" refers to an individual human being who appears in a news article or advertisement.

[0466] A "social networking service (SNS)" is an online platform that allows users to share information and communicate.

[0467] An "account" is user identification information required to use an SNS.

[0468] "Following" is the act of setting up a social networking site to receive updates from a specific user.

[0469] An "emotion engine" is a software technology for analyzing emotions from a user's facial expressions, voice, operation patterns, etc.

[0470] "Analysis" is the process of examining data in detail and understanding its structure and meaning.

[0471] "Follow Suggestions" are recommendations of social media accounts to follow that are displayed to users based on analyzed sentiment and other data.

[0472] "Advertisements" are promotional content such as images and videos created to advertise products or services.

[0473] "Image and video analysis" is the process of analyzing images and videos within an advertisement to identify people and objects contained within them.

[0474] This invention combines a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos in news articles and advertisements, allowing users to easily follow those accounts, with an emotion engine that recognizes the user's emotions. The details of the program for this system are described below.

[0475] Overall system overview

[0476] The system consists of three main components: a server, a terminal, and a user.

[0477] 1. Server:

[0478] The server receives image and video data from news applications and advertising videos and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It then uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow. The server uses OpenCV for image collection and analysis, and TensorFlow and Keras for machine learning models. It also uses Azure Face API and Google Cloud Vision API for emotion analysis.

[0479] 2. Terminal:

[0480] The device is a device on which the news application and advertisement display application operated by the user runs. The device displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, it sends the request to the server. The device must be equipped with a camera and microphone.

[0481] 3. User:

[0482] Users operate a news application or an advertising display application to view news articles or advertisements that interest them. They can easily follow the official social media accounts of people featured in images or videos in the articles or advertisements. They can also receive suggestions based on the analysis results of the emotion engine, allowing them to select more appropriate follow-up actions.

[0483] Specific examples

[0484] Scenario: User viewing an advertisement for a fashion brand

[0485] 1. A user views a fashion brand advertisement on their smartphone. At this time, the smartphone's camera captures the images and video data in the advertisement and sends them to a server.

[0486] 2. The server analyzes the received data using OpenCV and TensorFlow and extracts the features of the model appearing in the advertisement.

[0487] 3. The server uses Elasticsearch to match the features with the database to identify the person appearing in the advertisement. It then uses various social media APIs (e.g., Twitter API, Instagram Graph API) to send the identified model's official social media account information to the smartphone.

[0488] 4. The smartphone captures the user's facial expressions, voice, and operation patterns in real time and sends them to the server using the Azure Face API, Google Cloud Vision API, and Google Cloud Speech-to-Text API.

[0489] 5. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the smartphone.

[0490] 6. The smartphone displays the received information on the user interface, overlaid on the advertisement, and the user presses the follow button from the displayed account list and suggestions.

[0491] 7. The smartphone sends the click information of the follow button to the server. The server performs authentication procedures with each SNS based on the SNS authentication information and completes the follow process using the API.

[0492] 8. The server sends the follow-up processing results to the smartphone, and the smartphone displays a completion message to the user.

[0493] Prompt Sentence Examples

[0494] Design an application that allows users to easily follow the official social media accounts of models appearing in fashion brand advertisements while viewing them. The application will analyze images and videos from the advertisement to identify the models and search for their social media accounts. It will also analyze users' sentiment in real time and make follow suggestions based on that sentiment. Explain the necessary hardware and software, as well as the specific steps involved.

[0495] In this way, users can easily follow the official social media accounts of people featured in news articles and advertisements based on emotion-based suggestions.

[0496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0497] Step 1:

[0498] The user operates a news application or an advertisement display application on the device to view news articles or advertisements that interest them. The device acquires the displayed image or video data and sends it to the server.

[0499] Input: Image and video data of news articles and advertisements

[0500] Output: Send data to the server

[0501] How it works: Your smartphone's camera captures images and video data and sends it over the internet to a server.

[0502] Step 2:

[0503] The server analyzes the received image and video data using OpenCV and TensorFlow, and extracts the features of the people appearing in the images and videos.

[0504] Input: Image and video data

[0505] Output: Person features

[0506] How it works: The server analyzes the captured image and video data, inputs it into an AI model, and extracts the person's features.

[0507] Step 3:

[0508] The server uses Elasticsearch to match the extracted features with an internal database to identify the person in the photo. It then searches for the identified person's official social media accounts via the APIs of various social media platforms (e.g., Twitter API, Instagram Graph API) and extracts official account information.

[0509] Input: Person features

[0510] Output: Official SNS account information

[0511] How it works: The server compares the features with an internal database and searches for social media accounts based on the identified person's name.

[0512] Step 4:

[0513] The device captures the user's facial expressions, voice, and operation patterns in real time and sends this data to the server. The device's camera and microphone capture emotional data to identify the user's emotions.

[0514] Input: User's facial expressions, voice, and operation patterns

[0515] Output: Send emotion data to the server

[0516] How it works: The device captures emotional data in real time using a camera and microphone and sends it to a server.

[0517] Step 5:

[0518] The server uses an emotion engine (e.g., Azure Face API, Google Cloud Vision API, Google Cloud Speech-to-Text API) to analyze the user's emotions, and based on the results, creates follow suggestions and sends them to the device.

[0519] Input: Emotion data

[0520] Output: Follow suggestion information

[0521] How it works: The server uses the emotion engine to analyze the emotion data and generates follow-up suggestions based on the analysis results.

[0522] Step 6:

[0523] The device displays the received official SNS account information and sentiment analysis results on the user interface of the news application or advertisement display application, allowing the user to press the follow button.

[0524] Input: Follow suggestion information

[0525] Output: Display in the user interface

[0526] How it works: The device updates the user interface with the follow suggestion information and displays a follow button.

[0527] Step 7:

[0528] When a user presses the follow button, the device sends a follow request to the server, which also includes the user's social media authentication information.

[0529] Input: Follow button click information, SNS authentication information

[0530] Output: Send follow request to server

[0531] Operation: The device sends the follow button click information and SNS authentication information to the server.

[0532] Step 8:

[0533] The server receives the follow request and performs authentication procedures on each SNS platform based on the SNS authentication information. After successful authentication, the server uses the API to follow the target official account from the user's account. The device is notified of the follow processing results.

[0534] Input: Follow request, SNS authentication information

[0535] Output: Follow processing results

[0536] Operation: The server performs authentication procedures on each SNS platform and executes the follow process.

[0537] Step 9:

[0538] The terminal displays the follow-up processing result received from the server to the user as a completion message.

[0539] Input: Follow-up processing result

[0540] Output: Display of completion message

[0541] Operation: The device displays a completion message on the screen based on the results of the follow-up process.

[0542] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0543] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0544] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0545] [Second embodiment]

[0546] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0547] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0548] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0549] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0550] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0551] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0552] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0553] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0554] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0555] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0556] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0557] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0558] This invention provides a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles and allows users to easily follow those accounts. The program for this system is described in detail below.

[0559] Overall system overview

[0560] The system consists of three main components: a server, a terminal, and a user.

[0561] 1. Server

[0562] The server receives image and video data from the news application, identifies people using image recognition technology, searches for the identified people's official social media accounts on various social media platforms, and sends that information to the device.

[0563] 2. Terminal

[0564] The terminal is a device on which a news application operated by a user runs. The terminal displays the official SNS account information received from the server on a user interface, and when the user presses the follow button, the terminal sends the request to the server.

[0565] 3. Users

[0566] Users operate the news application to view news articles they are interested in. They can easily follow the official social media accounts of people featured in images and videos within the articles.

[0567] Program processing

[0568] 1. Image and video analysis

[0569] The device acquires image and video data from the news article the user is viewing and sends it to a server. The server then uses image recognition technology to identify people appearing in the images and videos based on the received data. The server then uses the latest AI models to extract features and compare them with a database to identify the person.

[0570] 2. Search for official social media accounts

[0571] After the person is identified, the server searches for the identified person's social media accounts on various social media platforms. Specifically, the server collects the official social media accounts via API based on the identified person's name and stores them in a database.

[0572] 3. Provide the user with account information

[0573] The server sends the search results to the device, which then displays the official social media account information within the news application, allowing users to view the official accounts of their interest on the news application interface.

[0574] 4. Implementing the Follow Feature

[0575] When a user presses the follow button from the displayed list of official accounts, the device sends the request to the server. The server automatically follows the official account based on the user's SNS authentication information. The server notifies the device of the completion of the follow and displays it to the user.

[0576] Specific examples

[0577] Scenario: A user views a news article featuring a famous actor.

[0578] 1. A user opens an article in a news application that features a famous actor.

[0579] 2. The device retrieves the image and video data from the article and sends it to the server.

[0580] 3. The server analyzes the received data using an "AI model" and extracts the actor's characteristics.

[0581] 4. The server compares the extracted features with a database to identify famous actors.

[0582] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[0583] 6. The device displays the received official account information in the news application.

[0584] 7. The user presses the follow button from the displayed list of official accounts.

[0585] 8. The device sends a request to the server that the follow button was pressed.

[0586] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[0587] 10. The server notifies the terminal of the results of the follow process, and the terminal displays a message to the user indicating that the follow has been completed.

[0588] This will create a system that allows users to easily follow the official social media accounts of people featured in news articles.

[0589] The processing flow will be explained below.

[0590] Step 1:

[0591] A user navigates through a news application and opens a news article of interest.

[0592] Step 2:

[0593] The device extracts image and video data contained in the news article and sends that data to the server.

[0594] Step 3:

[0595] The server inputs the received image and video data into the AI ​​model and extracts the features of the people appearing in the images and videos.

[0596] Step 4:

[0597] The server compares the extracted features with an internal database to identify the person being displayed.

[0598] Step 5:

[0599] Based on the information of the identified person, the server uses the APIs of various social media platforms to search for that person's official social media accounts.

[0600] Step 6:

[0601] The server identifies accounts that have official account marks (e.g., verification badges) and extracts this as official account information.

[0602] Step 7:

[0603] The server sends official SNS account information to the device.

[0604] Step 8:

[0605] The device displays the received official SNS account information on the interface within the news application.

[0606] Step 9:

[0607] The user clicks the follow button from the displayed list of official accounts.

[0608] Step 10:

[0609] When the user clicks the follow button, the device sends a follow request to the server, including the user's SNS authentication information.

[0610] Step 11:

[0611] The server receives the follow request and performs authentication procedures on each SNS platform based on the user's SNS authentication information.

[0612] Step 12:

[0613] After the server successfully authenticates the user, it uses the API to follow the target official account from the user's account.

[0614] Step 13:

[0615] The server notifies the terminal of the result of the follow-up process.

[0616] Step 14:

[0617] The device notifies the user by displaying a message in the news application indicating that the follow has been completed.

[0618] This series of processes allows users to easily follow the official social media accounts of people featured in news articles.

[0619] Example 1

[0620] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0621] Conventional news applications lack a means to quickly and accurately identify the official social networking service (SNS) accounts of people featured in articles and allow users to easily follow them. Furthermore, analyzing images and videos takes time, making it difficult to provide accurate account information.

[0622] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0623] In this invention, the server includes a means for a terminal to acquire and transmit image and video data of a news article, a means for the server to analyze the received image and video data and identify a person using image recognition technology, a means for the server to search for the identified person's official social networking service (SNS) account from various SNS platforms, a means for the server to perform a follow process based on the user's SNS authentication information, and a means for the terminal to display official SNS account information to the user and provide a follow button, thereby enabling the user to quickly and accurately identify the official SNS account of a person appearing in a news article and easily follow the person.

[0624] A "terminal" is a device on which a user operates a news application, and is generally an electronic device such as a smartphone, tablet, or PC.

[0625] "Images and video data of news articles" refers to visual content contained in articles provided within the news application, including still images and video data.

[0626] "Server" refers to a central processing unit that processes data received from a terminal and transmits analysis results and additional information to the terminal.

[0627] "Image recognition technology" refers to technology that uses computer vision techniques to identify people and objects in images and videos.

[0628] An "SNS platform" is an online system that provides social networking services, and generally refers to networks such as Twitter and Facebook.

[0629] An "AI model" refers to mathematical algorithms and data structures created based on artificial intelligence technology to automatically perform specific tasks (such as person identification).

[0630] "Features" refer to important attributes and patterns extracted from images and videos, and are used to identify people and objects.

[0631] "Official social media account" refers to an official social media account that is verified on a social media platform, such as the official profile of a celebrity or company.

[0632] A "follow button" is a button that is displayed on the user interface, and a user can click on it to follow a specific SNS account.

[0633] MODE FOR CARRYING OUT THE INVENTION

[0634] This invention provides a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles and enables users to easily follow those accounts. Detailed embodiments of this system are described below.

[0635] System Configuration

[0636] This system mainly consists of a server, terminals, and users.

[0637] 1. Server

[0638] The server receives image and video data sent from the news application and uses image recognition technology to identify people.

[0639] The server searches for the identified person's official social media account on various social media platforms and sends that information to the device.

[0640] The server is equipped with AI models such as TensorFlow and OpenCV, which are used to analyze the received data.

[0641] 2. Terminal

[0642] The terminal is a device on which a news application operated by a user runs, and includes a smartphone, tablet, PC, etc.

[0643] The terminal displays the official SNS account information received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[0644] 3. Users

[0645] The user operates the news application to view news articles of interest.

[0646] Users can easily follow the official social media accounts of people featured in images and videos within articles using the follow button.

[0647] Operation overview

[0648] Data Acquisition and Transmission

[0649] When a user uses a news application to view a news article, the device acquires the image and video data contained in the article and transmits it to a server.

[0650] Analysis and Identification

[0651] The server uses the latest AI models to extract features from the received image and video data, and compares them with a database to identify people, using image recognition technologies such as TensorFlow and OpenCV.

[0652] Account Search

[0653] The server searches for official accounts using the APIs of various social media platforms based on the identified person's name. The collected social media account information is temporarily stored in a database.

[0654] Information provision

[0655] The server sends the search results to the terminal, and the terminal displays the official SNS account information on the user interface.

[0656] Follow-up process

[0657] When a user presses the follow button, the device sends the request to the server. The server automatically follows the official account based on the user's SNS authentication information. The device is notified of the success or failure of this process, and displays a message to the user indicating that the follow has been completed.

[0658] Specific examples

[0659] Scenario: A user views a news article featuring a famous actor.

[0660] 1. A user opens an article in a news application that mentions a famous actor.

[0661] 2. The device retrieves the image and video data from the article and sends it to the server.

[0662] 3. The server analyzes the received data using TensorFlow and extracts the actor's features.

[0663] 4. The server compares the extracted features with a database to identify famous actors.

[0664] 5. The server searches for the official social media account using the identified actor's name using the "Facebook API" or "Twitter API" and sends the official account information to the device.

[0665] 6. The official account information received by the device is displayed within the news application.

[0666] 7. The user clicks the Follow button from the displayed list of official accounts.

[0667] 8. The device sends a request to the server that the follow button has been pressed.

[0668] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[0669] 10. The server notifies the device of the results of the follow process, and the device displays a message to the user indicating that the follow has been completed.

[0670] This system allows users to quickly and accurately identify the official social media accounts of people featured in news articles and easily follow them.

[0671] Example prompts to input to the generative AI model

[0672] prompt:

[0673] Describe a system that identifies the official social media accounts of celebrities mentioned in news articles. This system analyzes image and video data from news articles and provides users with the social media account information of the identified people, allowing them to easily follow them.

[0674] scenario:

[0675] 1. A user opens an article in a news application that features a famous actor.

[0676] 2. The device retrieves the image and video data from the article and sends it to the server.

[0677] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[0678] 4. The server compares the extracted features with a database to identify famous actors.

[0679] 5. The server uses an API to search for the official social media account for the identified actor's name and sends the official account information to the device.

[0680] 6. The device displays the received official account information in the news application.

[0681] 7. The user presses the follow button from the displayed list of official accounts.

[0682] 8. The device sends a request to the server that the follow button was pressed.

[0683] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[0684] 10. The server notifies the terminal of the results of the follow process, and the terminal displays a message to the user indicating that the follow has been completed.

[0685] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0686] Program processing flow

[0687] Step 1:

[0688] A user launches a news application and browses to a news article of interest.

[0689] Specifically, a user selects an article featuring a famous actor from the application's news feed, opens the article, and begins reading.

[0690] Input: User launches news application and selects article

[0691] Output: Display article content

[0692] Step 2:

[0693] The device acquires image and video data from the news article the user is viewing.

[0694] The device automatically extracts all image and video data contained in the article and temporarily stores it.

[0695] Input: News article data

[0696] Output: Extraction of image and video data

[0697] Step 3:

[0698] The image and video data acquired by the device is sent to the server.

[0699] The terminal collects all the extracted image and video data into packets and transmits them to the server.

[0700] Input: Image and video data

[0701] Output: Send data to the server

[0702] Step 4:

[0703] The image and video data received by the server is analyzed using an AI model (e.g., TensorFlow).

[0704] The server inputs the received data into an AI model and extracts the person's features, using facial recognition technology.

[0705] Input: Image and video data

[0706] Output: Feature extraction

[0707] Step 5:

[0708] The server identifies the person based on the extracted features by comparing them with a database.

[0709] The server compares the extracted features with a database of pre-trained people and identifies matching people.

[0710] Input: Feature data, database

[0711] Output: Identified person

[0712] Step 6:

[0713] The server searches for the official social media accounts of the identified person from various social media platforms (e.g., Facebook API, Twitter API).

[0714] The server uses the API of the social media platform to search for official accounts based on the identified person's name and collects that information.

[0715] Input: Name of the identified person

[0716] Output: Official social media account information

[0717] Step 7:

[0718] The server sends the collected official SNS account information to the device.

[0719] The server assembles the collected account information into packets and sends them to the terminal.

[0720] Input: Official SNS account information

[0721] Output: Send account information to the device

[0722] Step 8:

[0723] The SNS account information received by the device is displayed within the news application.

[0724] The device displays the received official SNS accounts in a list format on the user interface.

[0725] Input: Official SNS account information

[0726] Output: Display account information

[0727] Step 9:

[0728] The user clicks the follow button for the account of interest from the displayed list of official accounts.

[0729] The user selects the desired account from the displayed list and clicks the follow button.

[0730] Input: Official SNS account list

[0731] Output: Follow button click

[0732] Step 10:

[0733] The device sends a request to the server that the follow button has been pressed.

[0734] The terminal sends the user's follow request to the server, and the request includes the user's SNS authentication information.

[0735] Input: Follow request

[0736] Output: Request sent to server

[0737] Step 11:

[0738] The server follows the official account based on the user's social media authentication information.

[0739] The server executes the follow process using the SNS API based on the received request.

[0740] Input: Follow request, social media credentials

[0741] Output: Follow processing result

[0742] Step 12:

[0743] The server notifies the terminal of the follow-up processing result.

[0744] The server notifies the terminal of the success or failure of the follow process.

[0745] Input: Follow-up processing result

[0746] Output: Notification to terminal

[0747] Step 13:

[0748] The device displays a follow-up completion message to the user.

[0749] The terminal displays a message to the user indicating that the follow has been completed based on the result of the follow process received from the server.

[0750] Input: Notification from the server

[0751] Output: Display of follow-up completion message

[0752] (Application example 1)

[0753] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0754] There is a need for a system that can identify the official social media accounts of people appearing in images and videos displayed on web pages and allow users to easily follow them, but with existing technology it has been difficult to automate this process, and users have had to perform cumbersome operations. There is a need to solve these issues and improve the user experience.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0756] In this invention, the server includes means for identifying a person appearing in an image or video on a webpage using image recognition technology, means for searching for the official information acquisition service account of the identified person, means for allowing a user to follow the official information acquisition service account within the website, means for displaying the official information acquisition service account information of the identified person on a user interface, and means for automatically following the official information acquisition service account based on the account information acquired by the user. This allows a user to easily follow a person's official SNS account without performing complicated operations.

[0757] "Image recognition technology" is a technology that extracts useful information from video data and identifies people and objects.

[0758] A "web page" is a document containing information that is viewable on the Internet.

[0759] "Images and videos" refers to still images and moving video data that contain visual information.

[0760] "People" refers to people appearing in an image or video.

[0761] An "official information acquisition service account" is an account that is authenticated on a social networking service (SNS).

[0762] "Searching" is the process of locating specific information.

[0763] A "user" is a person who uses the system.

[0764] A "user interface" is a screen and operating means for a user to interact with a system.

[0765] "Following" refers to linking with a specific account on social media and receiving information.

[0766] A "server" is a computer system that processes data on a network.

[0767] The system for implementing this invention consists of three main components: a server, a terminal, and a user.

[0768] 1. Server Processing

[0769] The server receives image and video data from web pages and uses image recognition technology to identify people. Software such as TensorFlow and PIL (Python Imaging Library) is installed on the server, which processes and calculates the received image and video data. Specifically, features are extracted from the image data and a generative AI model is used to match the data with a database to identify the person. The identified person's official social media account is searched for via the API of each social media platform. The search results are saved for subsequent processing.

[0770] 2. Terminal Processing

[0771] The terminal is a device that displays the web page operated by the user, and has the role of displaying the account information of the official information acquisition service received from the server on the user interface. When the user presses the follow button, the request is sent from the terminal to the server.

[0772] 3. User Operation

[0773] While browsing a website, users may want to follow the official information service account of a person who appears in an image or video on the page. In this case, users can easily follow the official account by pressing the follow button displayed on their device.

[0774] Specific examples

[0775] For example, suppose a user is browsing a favorite product page on a fashion shopping site. The product page features an image of a famous model. Using this system, the following happens:

[0776] 1. When a user opens a product page, the device sends the image data of that page to the server.

[0777] 2. The server uses TensorFlow to perform image analysis and extract model features.

[0778] 3. Based on the features, the server matches the database and identifies the model.

[0779] 4. The server searches for the identified model's official information retrieval service account (e.g., Instagram or Twitter) using the API of each social media platform.

[0780] 5. The retrieved account information is sent to the device and displayed on the user's web page.

[0781] 6. Users can follow the model's official account from their own social media account by clicking the follow button that appears.

[0782] Prompt Sentence Examples

[0783] Example prompts to input to generative AI models such as Chat GPT:

[0784] We are developing a system that uses image analysis technology to identify the official social media accounts of people featured in specific news articles and automatically follow those accounts. Please generate a prompt sentence under the following conditions:

[0785] The system first retrieves an image / video of the page the user is viewing.

[0786] The captured images / videos are sent to the server and people are identified using TensorFlow.

[0787] Search for the official social media accounts of the identified person using the social media API.

[0788] Displays SNS account information to users. Users can follow official accounts by clicking the follow button.

[0789] Finally, we display a message to the user confirming that they have followed us.

[0790] This allows a system to be realized that allows users to easily follow a person's official SNS account when browsing a web page.

[0791] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0792] Step 1:

[0793] The device acquires image and video data from the web page the user is viewing. The URL of the web page and a reference image are given as input. The device then sends the acquired image data to the server. In this step, the device collects image data using a browser or a specific application installed on the device, converts the received data into an appropriate format, and passes it to the server.

[0794] Step 2:

[0795] The server processes the image and video data received from the device. First, it uses image recognition technology to extract the features of the people in the image. Image data is given as input, and the person's identification information is obtained as output. Then, TensorFlow is used to perform image analysis and extract specific features (such as facial contours and the placement of the eyes, nose, and mouth). This generates data that can be used to identify specific people in the image or video.

[0796] Step 3:

[0797] The server uses the extracted features to match the database and identify the person. Feature data is given as input, and the identified person's identity is obtained as output. The server uses a generative AI model to quickly search the existing database and return the corresponding person information. This step identifies who the person in the image is.

[0798] Step 4:

[0799] The server searches each SNS platform for the identified person's official information retrieval service account. The person's identification information is given as input, and SNS account information is obtained as output. The server uses the SNS API to send a search query based on the identified person's name and collects official account information. In this step, only authenticated accounts are extracted and listed.

[0800] Step 5:

[0801] The server sends the identified official information retrieval service account information to the device. The social media account information is given as input, and account information displayed in the user interface is generated as output. The server sends formatted data so that the information can be displayed correctly on the device. This data includes the official mark, account name, and profile URL.

[0802] Step 6:

[0803] The device displays the received official information acquisition service account information on a user interface. The account information sent from the server is given as input, and information that can be visually confirmed by the user is displayed as output. The device appropriately arranges the information on a web page and provides interactive elements including a follow button.

[0804] Step 7:

[0805] The user presses the Follow button from the displayed list of official information acquisition service accounts. The user's action is taken as input, and a follow request is generated as output. In this step, the Follow button responds to the user's action as a trigger, and the next action is taken.

[0806] Step 8:

[0807] When the user presses the follow button, the device sends a follow request to the server. User operation information is given as input, and the follow request is sent to the server as output. The device generates follow request data and makes a specific API call to the server.

[0808] Step 9:

[0809] The server receives a follow request from a user and automatically follows a specific official information retrieval service account based on that information. The follow request information is given as input, and a follow completion status is generated as output. The server executes the follow action through the SNS API.

[0810] Step 10:

[0811] The server notifies the device of the results of the follow process, and the device displays a message to the user indicating that the follow has been completed. The follow completion information is given as input, and a confirmation message is generated as output and displayed to the user. The server and device work together to allow the user to confirm that the follow was successful.

[0812] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0813] This invention combines a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles, allowing users to easily follow those accounts, with an emotion engine that recognizes the user's emotions. The program for this system is described in detail below.

[0814] Overall system overview

[0815] The system consists of three main components: a server, a terminal, and a user.

[0816] 1. Server

[0817] The server receives image and video data from the news application and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It also uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow.

[0818] 2. Terminal

[0819] The terminal is a device that runs a news application operated by the user. The terminal displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[0820] 3. Users

[0821] Users operate the news application to browse news articles of interest. They can easily follow the official social media accounts of people featured in images and videos within the article. They can also select more appropriate follow-up actions by receiving suggestions based on the analysis results of the emotion engine.

[0822] Program processing

[0823] 1. Image and video analysis

[0824] When a user operates a news application and opens a news article, the device retrieves the images and video data from the article and sends them to the server. The server then inputs the received image and video data into an AI model and extracts the features of the people appearing in the images and videos.

[0825] 2. Identifying people and searching social media accounts

[0826] The server identifies the person displayed by checking the extracted features against an internal database. It then searches for the identified person's official social media accounts via the APIs of various social media platforms, identifies accounts with the official account mark (verification badge), and extracts this as official account information.

[0827] 3. User Emotion Recognition and Analysis

[0828] The device captures the user's facial expressions, voice, operation patterns, etc. in real time and sends this data to the server. The server uses an emotion engine to analyze this data and recognize the user's emotional state. Emotional information indicates the user's basic emotional state, such as joy, anger, sadness, and happiness.

[0829] 4. Providing account information and sentiment analysis results

[0830] The server creates suggestions for following SNS accounts that match the user's emotions based on the official SNS account information and the results of the user's emotion analysis, and sends the suggestions to the device. The device then displays the received information on the interface within the news application.

[0831] 5. Implementing the Follow Feature

[0832] The user presses the follow button from the list of official accounts displayed and suggestions based on the sentiment analysis results. When the follow button is pressed, the device sends a follow request to the server. This request also includes the user's SNS authentication information.

[0833] 6. Execute follow-up process

[0834] The server receives the follow request and performs authentication procedures on each SNS platform based on the SNS authentication information. After successful authentication, the API is used to follow the target official account from the user's account. The server notifies the device of the follow processing results, and the device displays a message to the user indicating that the follow has been completed.

[0835] Specific examples

[0836] Scenario: A user views a news article featuring a famous actor.

[0837] 1. A user opens an article in a news application that mentions a famous actor.

[0838] 2. The device retrieves the image and video data from the article and sends it to the server.

[0839] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[0840] 4. The server compares the features with the database to identify famous actors.

[0841] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[0842] 6. The device acquires the user's facial expressions, voice, and operation patterns and sends them to the server.

[0843] 7. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the device.

[0844] 8. The device displays the received information within the app.

[0845] 9. The user presses the Follow button from the displayed list of accounts and suggestions.

[0846] 10. The device sends the follow button click information to the server.

[0847] 11. The server performs authentication procedures on each SNS based on the SNS authentication information and completes the follow process using the API.

[0848] 12. The server sends the results of the follow process to the terminal, and the terminal displays a message to the user indicating that the follow has been completed.

[0849] This system allows users to easily follow the official social media accounts of people featured in news articles based on sentiment-based suggestions.

[0850] The processing flow will be explained below.

[0851] Step 1:

[0852] A user navigates through a news application and opens a news article of interest.

[0853] Step 2:

[0854] The device acquires image and video data from the article and sends that data to the server.

[0855] Step 3:

[0856] The server inputs the received image and video data into the AI ​​model and extracts the features of the people appearing in the images and videos.

[0857] Step 4:

[0858] The server compares the extracted features with an internal database to identify the person being displayed.

[0859] Step 5:

[0860] Based on the information of the identified person, the server uses the APIs of various social media platforms to search for that person's official social media accounts.

[0861] Step 6:

[0862] The server identifies accounts that have official account marks (e.g., verification badges) and extracts this as official account information.

[0863] Step 7:

[0864] The terminal acquires data such as the user's facial expressions, voice, and operation patterns, and sends this data to the server.

[0865] Step 8:

[0866] The server inputs the received data into the emotion engine and analyzes the user's emotional state. For example, if the user smiles, it identifies the emotional state as "joy," and if the user frowns, it identifies the emotional state as "dissatisfaction."

[0867] Step 9:

[0868] Based on the results of the emotion analysis, the server creates suggestions for following official social media accounts that match the user's emotions.

[0869] Step 10:

[0870] The server sends official SNS account information and emotion analysis results to the device.

[0871] Step 11:

[0872] The device will display the received official social media account information and follow suggestions on the interface within the news application.

[0873] Step 12:

[0874] Users can click the follow button from the displayed list of official accounts and suggestions based on sentiment analysis results.

[0875] Step 13:

[0876] When the user clicks the follow button, the device sends a follow request to the server, including the user's SNS authentication information.

[0877] Step 14:

[0878] The server receives the follow request and performs authentication procedures on each SNS platform based on the user's SNS authentication information.

[0879] Step 15:

[0880] After successful authentication, the server uses the API to follow the target official account from the user's account.

[0881] Step 16:

[0882] The server notifies the terminal of the result of the follow-up process.

[0883] Step 17:

[0884] The device notifies the user by displaying a message in the news application indicating that the follow has been completed.

[0885] Example 2

[0886] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0887] In conventional news applications, when users want to follow the official social networking service (SNS) accounts of people featured in news articles, they must manually search for them, which is time-consuming. Furthermore, there is insufficient information to determine whether the account is of interest to the user, and simply providing account information does not sufficiently stimulate the user's interest. This not only requires time and effort to follow, but also results in a lack of improvement in the user experience.

[0888] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0889] In this invention, the server includes means for identifying people appearing in images and videos of news articles using image recognition technology, means for searching for the identified people's official social networking service (SNS) accounts, means for allowing users to follow the official SNS accounts within the news application, means for acquiring the user's emotional data in real time, means for analyzing the user's emotional state using an emotion engine, and means for creating follow suggestions based on the analyzed emotional information. This allows users to easily follow the official SNS accounts of people appearing in news articles, and further provides a more engaging experience by receiving appropriate follow suggestions based on emotions.

[0890] "Image recognition technology" is a technology that identifies and analyzes specific features and patterns from digital images and videos.

[0891] A "news article" refers to the informational content published in the media, including multimedia content such as text, images, and video.

[0892] "Person identification" is the process of identifying people in images or videos using specific methods and algorithms.

[0893] A "social networking service (SNS)" is an online platform that enables users to communicate with each other over the Internet.

[0894] An "official SNS account" is an account officially authenticated by an individual or organization on a social networking service.

[0895] "Following" is the act of registering another user on a social networking site with the aim of receiving their posts and information on a regular basis.

[0896] "Emotion data" is data that indicates the user's emotional state, and is obtained from facial expressions, voice, operation patterns, and the like.

[0897] An "emotion engine" is software or an algorithm that analyzes a user's emotional state based on collected emotional data.

[0898] "Follow suggestions" are suggestions that recommend social media accounts for users to follow based on analyzed information and the user's interests.

[0899] A "server" is a computer system that processes data and provides information in response to user requests.

[0900] A "terminal" is a device that allows a user to operate a news application, and includes a smartphone, tablet, or the like.

[0901] This invention is a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles, allowing users to easily follow those accounts, and combines this with an emotion engine that recognizes the user's emotions. This system consists of three main components: a server, a terminal, and a user.

[0902] Overall system overview

[0903] 1. Server

[0904] The server receives image and video data from the news application and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It then uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow.

[0905] 2. Terminal

[0906] The terminal is a device that runs a news application operated by the user. The terminal displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[0907] 3. Users

[0908] Users operate the news application to browse news articles of interest. They can easily follow the official social media accounts of people featured in images and videos within the article. They can also select more appropriate follow-up actions by receiving suggestions based on the analysis results of the emotion engine.

[0909] Hardware and software used

[0910] The server is a server machine equipped with a high-performance processor that runs an AI model using Python (e.g., TensorFlow, OpenCV). The emotion engine uses an emotion analysis service such as Microsoft Azure Emotion API. The terminal is a device such as a smartphone or tablet that exchanges data with the server in real time using WebSocket technology. The terminal is also equipped with a camera and microphone and has the ability to capture the user's facial expressions and voice.

[0911] Specific examples

[0912] Scenario: A user views a news article featuring a famous actor.

[0913] 1. A user opens an article in a news application that mentions a famous actor.

[0914] 2. The device retrieves the image and video data from the article and sends it to the server.

[0915] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[0916] 4. The server compares the features with the database to identify famous actors.

[0917] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[0918] 6. The device acquires the user's facial expressions, voice, and operation patterns and sends them to the server.

[0919] 7. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the device.

[0920] 8. The device displays the received information within the app.

[0921] 9. The user presses the Follow button from the displayed list of accounts and suggestions.

[0922] 10. The device sends the follow button click information to the server.

[0923] 11. The server performs authentication procedures on each SNS based on the SNS authentication information and completes the follow process using the API.

[0924] 12. The server sends the results of the follow process to the terminal, and the terminal displays a message to the user indicating that the follow has been completed.

[0925] Prompt Sentence Examples

[0926] "Please explain how you input image data into your AI model and extract features of people in the image. Please be specific about the process, including the specific libraries and tools you use."

[0927] This system allows users to easily follow the official social media accounts of people featured in news articles based on sentiment-based suggestions.

[0928] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0929] Step 1:

[0930] A user launches a news application and selects a particular news article, which includes images and video.

[0931] Input: User action to select a news article

[0932] Output: Start retrieving image and video data from articles

[0933] What happens: A user opens a news application on their smartphone and clicks on an article of interest from the news feed.

[0934] Step 2:

[0935] The device acquires image and video data from the article, and sends the acquired data to the server.

[0936] Input: URL of image or video in news article

[0937] Output: Binary image and video data

[0938] Specific operation: The device extracts the URLs of images and videos from the DOM of the article displayed by the news application, and sends the obtained image and video data in binary format to the server.

[0939] Step 3:

[0940] The server inputs the received image and video data into the AI ​​model and extracts the features of the people in the images.

[0941] Input: binary image or video data

[0942] Output: Extracted person feature data

[0943] How it works: The server uses Python libraries (e.g., TensorFlow, OpenCV) to input image data into the AI ​​model, and the model extracts facial features (eye position, nose shape, jaw line, etc.).

[0944] Step 4:

[0945] The server compares the extracted features with an internal database to identify the person being displayed.

[0946] Input: Extracted person feature data

[0947] Output: Data of identified person

[0948] Specific operation: The server compares the extracted feature data with an internal database and obtains the ID and name of the identified person.

[0949] Step 5:

[0950] The server searches for the identified person's official social media accounts using the APIs of various social media platforms, extracts official accounts (with verification badges) from the search results, and sends that information to the device.

[0951] Input: ID or name of the identified person

[0952] Output: Official social media account information (with verification badge)

[0953] Specific operation: The server uses the Twitter API or Facebook Graph API to search for official social media accounts using the identified person's name, extracts official account information with a verification badge from the obtained account data, and sends it to the device in JSON format.

[0954] Step 6:

[0955] The device acquires emotional data such as the user's facial expressions, voice, and operation patterns in real time and transmits this data to the server.

[0956] Input: User's facial expression, voice, and operation pattern data

[0957] Output: Collected emotion data

[0958] Specific operation: The device uses a camera and microphone to capture the user's facial expressions and voice, and also records operation data such as the user's touch pattern and scrolling speed, and sends it to the server via WebSocket.

[0959] Step 7:

[0960] The server uses an emotion engine to analyze the received emotion data and recognize the user's emotional state. Based on the results, it creates suggestions for following SNS accounts suitable for the user and sends them to the device.

[0961] Input: Collected emotion data

[0962] Output: Sentiment analysis results and follow suggestion data

[0963] How it works: The server uses an emotion analysis library (e.g., Microsoft Azure Emotion API) to analyze facial expressions and voice data to identify the user's emotional state (joy, anger, sadness, etc.), and then generates follow suggestions based on this and sends the information to the device.

[0964] Step 8:

[0965] The device displays the received SNS account information and emotion analysis results on the user interface and sends a request to the server for the user to press the follow button.

[0966] Input: Follow suggestion data

[0967] Output: User's follow button operation information

[0968] Specific operation: The device displays official social media account information and sentiment analysis results on the interface within the news app, receives instructions from the user to press the follow button for accounts that interest them, and sends this information to the server.

[0969] Step 9:

[0970] The server receives the click information of the follow button, performs authentication procedures with the corresponding SNS based on the SNS authentication information, and then completes the follow process using the API.

[0971] Input: User follow button operation information, SNS authentication information

[0972] Output: Follow processing result

[0973] Specific operation: The server uses the SNS API to send a follow request based on the user's SNS authentication information and receives a response indicating whether the follow process was successful.

[0974] Step 10:

[0975] The server notifies the terminal of the result of the follow process, and the terminal displays a message indicating that the follow has been completed to the user.

[0976] Input: Follow-up processing result data

[0977] Output: Follow completion message

[0978] Specific operation: The server sends data to the device indicating that the follow process has been completed, and the device then pops up a "Follow Complete" message to the user based on that data.

[0979] This series of processes allows users to easily follow the official social media accounts of people featured in news articles, and by receiving appropriate follow suggestions based on their emotions, users can have a more engaging experience.

[0980] (Application example 2)

[0981] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0982] Conventional news applications have the drawback of requiring users to identify and follow the social media accounts of people featured in articles, and of not suggesting appropriate accounts to follow that reflect the user's interests. Furthermore, in the advertising field, there is a lack of functionality that allows users to easily identify the social media accounts of people featured in advertisements and follow them based on their emotions. Therefore, there is a need for a method to increase user engagement and maximize advertising effectiveness.

[0983] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0984] In this invention, the server includes means for identifying people appearing in images and videos of news articles using image recognition technology, means for searching for the official social networking service (SNS) accounts of the identified people, means for analyzing the user's emotions using an emotion engine that recognizes the user's emotions, means for suggesting appropriate official SNS accounts to follow based on the analyzed emotions, and means for identifying people in advertisements based on image and video analysis of the advertisements and searching for their SNS accounts. This enables users to easily follow the official SNS accounts of people appearing in news articles and advertisements based on emotion-based suggestions.

[0985] "Image recognition technology" is a technology for identifying and analyzing people and objects from images and video data acquired using cameras and sensors.

[0986] A "news article" is a document or content that describes news reports published in newspapers, websites, etc.

[0987] A "video" is a medium that expresses movement by playing back a series of multiple images along a time axis.

[0988] "Person" refers to an individual human being who appears in a news article or advertisement.

[0989] A "social networking service (SNS)" is an online platform that allows users to share information and communicate.

[0990] An "account" is user identification information required to use an SNS.

[0991] "Following" is the act of setting up a social networking site to receive updates from a specific user.

[0992] An "emotion engine" is a software technology for analyzing emotions from a user's facial expressions, voice, operation patterns, etc.

[0993] "Analysis" is the process of examining data in detail and understanding its structure and meaning.

[0994] "Follow Suggestions" are recommendations of social media accounts to follow that are displayed to users based on analyzed sentiment and other data.

[0995] "Advertisements" are promotional content such as images and videos created to advertise products or services.

[0996] "Image and video analysis" is the process of analyzing images and videos within an advertisement to identify people and objects contained within them.

[0997] This invention combines a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos in news articles and advertisements, allowing users to easily follow those accounts, with an emotion engine that recognizes the user's emotions. The details of the program for this system are described below.

[0998] Overall system overview

[0999] The system consists of three main components: a server, a terminal, and a user.

[1000] 1. Server:

[1001] The server receives image and video data from news applications and advertising videos and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It then uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow. The server uses OpenCV for image collection and analysis, and TensorFlow and Keras for machine learning models. It also uses Azure Face API and Google Cloud Vision API for emotion analysis.

[1002] 2. Terminal:

[1003] The device is a device on which the news application and advertisement display application operated by the user runs. The device displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, it sends the request to the server. The device must be equipped with a camera and microphone.

[1004] 3. User:

[1005] Users operate a news application or an advertising display application to view news articles or advertisements that interest them. They can easily follow the official social media accounts of people featured in images or videos in the articles or advertisements. They can also receive suggestions based on the analysis results of the emotion engine, allowing them to select more appropriate follow-up actions.

[1006] Specific examples

[1007] Scenario: User viewing an advertisement for a fashion brand

[1008] 1. A user views a fashion brand advertisement on their smartphone. At this time, the smartphone's camera captures the images and video data in the advertisement and sends them to a server.

[1009] 2. The server analyzes the received data using OpenCV and TensorFlow and extracts the features of the model appearing in the advertisement.

[1010] 3. The server uses Elasticsearch to match the features with the database to identify the person appearing in the advertisement. It then uses various social media APIs (e.g., Twitter API, Instagram Graph API) to send the identified model's official social media account information to the smartphone.

[1011] 4. The smartphone captures the user's facial expressions, voice, and operation patterns in real time and sends them to the server using the Azure Face API, Google Cloud Vision API, and Google Cloud Speech-to-Text API.

[1012] 5. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the smartphone.

[1013] 6. The smartphone displays the received information on the user interface, overlaid on the advertisement, and the user presses the follow button from the displayed account list and suggestions.

[1014] 7. The smartphone sends the click information of the follow button to the server. The server performs authentication procedures with each SNS based on the SNS authentication information and completes the follow process using the API.

[1015] 8. The server sends the follow-up processing results to the smartphone, and the smartphone displays a completion message to the user.

[1016] Prompt Sentence Examples

[1017] Design an application that allows users to easily follow the official social media accounts of models appearing in fashion brand advertisements while viewing them. The application will analyze images and videos from the advertisement to identify the models and search for their social media accounts. It will also analyze users' sentiment in real time and make follow suggestions based on that sentiment. Explain the necessary hardware and software, as well as the specific steps involved.

[1018] In this way, users can easily follow the official social media accounts of people featured in news articles and advertisements based on emotion-based suggestions.

[1019] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1020] Step 1:

[1021] The user operates a news application or an advertisement display application on the device to view news articles or advertisements that interest them. The device acquires the displayed image or video data and sends it to the server.

[1022] Input: Image and video data of news articles and advertisements

[1023] Output: Send data to the server

[1024] How it works: Your smartphone's camera captures images and video data and sends it over the internet to a server.

[1025] Step 2:

[1026] The server analyzes the received image and video data using OpenCV and TensorFlow, and extracts the features of the people appearing in the images and videos.

[1027] Input: Image and video data

[1028] Output: Person features

[1029] How it works: The server analyzes the captured image and video data, inputs it into an AI model, and extracts the person's features.

[1030] Step 3:

[1031] The server uses Elasticsearch to match the extracted features with an internal database to identify the person in the photo. It then searches for the identified person's official social media accounts via the APIs of various social media platforms (e.g., Twitter API, Instagram Graph API) and extracts official account information.

[1032] Input: Person features

[1033] Output: Official SNS account information

[1034] How it works: The server compares the features with an internal database and searches for social media accounts based on the identified person's name.

[1035] Step 4:

[1036] The device captures the user's facial expressions, voice, and operation patterns in real time and sends this data to the server. The device's camera and microphone capture emotional data to identify the user's emotions.

[1037] Input: User's facial expressions, voice, and operation patterns

[1038] Output: Send emotion data to the server

[1039] How it works: The device captures emotional data in real time using a camera and microphone and sends it to a server.

[1040] Step 5:

[1041] The server uses an emotion engine (e.g., Azure Face API, Google Cloud Vision API, Google Cloud Speech-to-Text API) to analyze the user's emotions, and based on the results, creates follow suggestions and sends them to the device.

[1042] Input: Emotion data

[1043] Output: Follow suggestion information

[1044] How it works: The server uses the emotion engine to analyze the emotion data and generates follow-up suggestions based on the analysis results.

[1045] Step 6:

[1046] The device displays the received official SNS account information and sentiment analysis results on the user interface of the news application or advertisement display application, allowing the user to press the follow button.

[1047] Input: Follow suggestion information

[1048] Output: Display in the user interface

[1049] How it works: The device updates the user interface with the follow suggestion information and displays a follow button.

[1050] Step 7:

[1051] When a user presses the follow button, the device sends a follow request to the server, which also includes the user's social media authentication information.

[1052] Input: Follow button click information, SNS authentication information

[1053] Output: Send follow request to server

[1054] Operation: The device sends the follow button click information and SNS authentication information to the server.

[1055] Step 8:

[1056] The server receives the follow request and performs authentication procedures on each SNS platform based on the SNS authentication information. After successful authentication, the server uses the API to follow the target official account from the user's account. The device is notified of the follow processing results.

[1057] Input: Follow request, SNS authentication information

[1058] Output: Follow processing results

[1059] Operation: The server performs authentication procedures on each SNS platform and executes the follow process.

[1060] Step 9:

[1061] The terminal displays the follow-up processing result received from the server to the user as a completion message.

[1062] Input: Follow-up processing result

[1063] Output: Display of completion message

[1064] Operation: The device displays a completion message on the screen based on the results of the follow-up process.

[1065] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1066] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1067] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1068] [Third embodiment]

[1069] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1070] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1071] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1072] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1073] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1074] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1075] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1076] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1077] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1078] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1079] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1080] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1081] This invention provides a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles and allows users to easily follow those accounts. The program for this system is described in detail below.

[1082] Overall system overview

[1083] The system consists of three main components: a server, a terminal, and a user.

[1084] 1. Server

[1085] The server receives image and video data from the news application, identifies people using image recognition technology, searches for the identified people's official social media accounts on various social media platforms, and sends that information to the device.

[1086] 2. Terminal

[1087] The terminal is a device on which a news application operated by a user runs. The terminal displays the official SNS account information received from the server on a user interface, and when the user presses the follow button, the terminal sends the request to the server.

[1088] 3. Users

[1089] Users operate the news application to view news articles they are interested in. They can easily follow the official social media accounts of people featured in images and videos within the articles.

[1090] Program processing

[1091] 1. Image and video analysis

[1092] The device acquires image and video data from the news article the user is viewing and sends it to a server. The server then uses image recognition technology to identify people appearing in the images and videos based on the received data. The server then uses the latest AI models to extract features and compare them with a database to identify the person.

[1093] 2. Search for official social media accounts

[1094] After the person is identified, the server searches for the identified person's social media accounts on various social media platforms. Specifically, the server collects the official social media accounts via API based on the identified person's name and stores them in a database.

[1095] 3. Provide the user with account information

[1096] The server sends the search results to the device, which then displays the official social media account information within the news application, allowing users to view the official accounts of their interest on the news application interface.

[1097] 4. Implementing the Follow Feature

[1098] When a user presses the follow button from the displayed list of official accounts, the device sends the request to the server. The server automatically follows the official account based on the user's SNS authentication information. The server notifies the device of the completion of the follow and displays it to the user.

[1099] Specific examples

[1100] Scenario: A user views a news article featuring a famous actor.

[1101] 1. A user opens an article in a news application that features a famous actor.

[1102] 2. The device retrieves the image and video data from the article and sends it to the server.

[1103] 3. The server analyzes the received data using an "AI model" and extracts the actor's characteristics.

[1104] 4. The server compares the extracted features with a database to identify famous actors.

[1105] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[1106] 6. The device displays the received official account information in the news application.

[1107] 7. The user presses the follow button from the displayed list of official accounts.

[1108] 8. The device sends a request to the server that the follow button was pressed.

[1109] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[1110] 10. The server notifies the terminal of the results of the follow process, and the terminal displays a message to the user indicating that the follow has been completed.

[1111] This will create a system that allows users to easily follow the official social media accounts of people featured in news articles.

[1112] The processing flow will be explained below.

[1113] Step 1:

[1114] A user navigates through a news application and opens a news article of interest.

[1115] Step 2:

[1116] The device extracts image and video data contained in the news article and sends that data to the server.

[1117] Step 3:

[1118] The server inputs the received image and video data into the AI ​​model and extracts the features of the people appearing in the images and videos.

[1119] Step 4:

[1120] The server compares the extracted features with an internal database to identify the person being displayed.

[1121] Step 5:

[1122] Based on the information of the identified person, the server uses the APIs of various social media platforms to search for that person's official social media accounts.

[1123] Step 6:

[1124] The server identifies accounts that have official account marks (e.g., verification badges) and extracts this as official account information.

[1125] Step 7:

[1126] The server sends official SNS account information to the device.

[1127] Step 8:

[1128] The device displays the received official SNS account information on the interface within the news application.

[1129] Step 9:

[1130] The user clicks the follow button from the displayed list of official accounts.

[1131] Step 10:

[1132] When the user clicks the follow button, the device sends a follow request to the server, including the user's SNS authentication information.

[1133] Step 11:

[1134] The server receives the follow request and performs authentication procedures on each SNS platform based on the user's SNS authentication information.

[1135] Step 12:

[1136] After the server successfully authenticates the user, it uses the API to follow the target official account from the user's account.

[1137] Step 13:

[1138] The server notifies the terminal of the result of the follow-up process.

[1139] Step 14:

[1140] The device notifies the user by displaying a message in the news application indicating that the follow has been completed.

[1141] This series of processes allows users to easily follow the official social media accounts of people featured in news articles.

[1142] Example 1

[1143] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1144] Conventional news applications lack a means to quickly and accurately identify the official social networking service (SNS) accounts of people featured in articles and allow users to easily follow them. Furthermore, analyzing images and videos takes time, making it difficult to provide accurate account information.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1146] In this invention, the server includes a means for a terminal to acquire and transmit image and video data of a news article, a means for the server to analyze the received image and video data and identify a person using image recognition technology, a means for the server to search for the identified person's official social networking service (SNS) account from various SNS platforms, a means for the server to perform a follow process based on the user's SNS authentication information, and a means for the terminal to display official SNS account information to the user and provide a follow button, thereby enabling the user to quickly and accurately identify the official SNS account of a person appearing in a news article and easily follow the person.

[1147] A "terminal" is a device on which a user operates a news application, and is generally an electronic device such as a smartphone, tablet, or PC.

[1148] "Images and video data of news articles" refers to visual content contained in articles provided within the news application, including still images and video data.

[1149] "Server" refers to a central processing unit that processes data received from a terminal and transmits analysis results and additional information to the terminal.

[1150] "Image recognition technology" refers to technology that uses computer vision techniques to identify people and objects in images and videos.

[1151] An "SNS platform" is an online system that provides social networking services, and generally refers to networks such as Twitter and Facebook.

[1152] An "AI model" refers to mathematical algorithms and data structures created based on artificial intelligence technology to automatically perform specific tasks (such as person identification).

[1153] "Features" refer to important attributes and patterns extracted from images and videos, and are used to identify people and objects.

[1154] "Official social media account" refers to an official social media account that is verified on a social media platform, such as the official profile of a celebrity or company.

[1155] A "follow button" is a button that is displayed on the user interface, and a user can click on it to follow a specific SNS account.

[1156] MODE FOR CARRYING OUT THE INVENTION

[1157] This invention provides a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles and enables users to easily follow those accounts. Detailed embodiments of this system are described below.

[1158] System Configuration

[1159] This system mainly consists of a server, terminals, and users.

[1160] 1. Server

[1161] The server receives image and video data sent from the news application and uses image recognition technology to identify people.

[1162] The server searches for the identified person's official social media account on various social media platforms and sends that information to the device.

[1163] The server is equipped with AI models such as TensorFlow and OpenCV, which are used to analyze the received data.

[1164] 2. Terminal

[1165] The terminal is a device on which a news application operated by a user runs, and includes a smartphone, tablet, PC, etc.

[1166] The terminal displays the official SNS account information received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[1167] 3. Users

[1168] The user operates the news application to view news articles of interest.

[1169] Users can easily follow the official social media accounts of people featured in images and videos within articles using the follow button.

[1170] Operation overview

[1171] Data Acquisition and Transmission

[1172] When a user uses a news application to view a news article, the device acquires the image and video data contained in the article and transmits it to a server.

[1173] Analysis and Identification

[1174] The server uses the latest AI models to extract features from the received image and video data, and compares them with a database to identify people, using image recognition technologies such as TensorFlow and OpenCV.

[1175] Account Search

[1176] The server searches for official accounts using the APIs of various social media platforms based on the identified person's name. The collected social media account information is temporarily stored in a database.

[1177] Information provision

[1178] The server sends the search results to the terminal, and the terminal displays the official SNS account information on the user interface.

[1179] Follow-up process

[1180] When a user presses the follow button, the device sends the request to the server. The server automatically follows the official account based on the user's SNS authentication information. The device is notified of the success or failure of this process, and displays a message to the user indicating that the follow has been completed.

[1181] Specific examples

[1182] Scenario: A user views a news article featuring a famous actor.

[1183] 1. A user opens an article in a news application that mentions a famous actor.

[1184] 2. The device retrieves the image and video data from the article and sends it to the server.

[1185] 3. The server analyzes the received data using TensorFlow and extracts the actor's features.

[1186] 4. The server compares the extracted features with a database to identify famous actors.

[1187] 5. The server searches for the official social media account using the identified actor's name using the "Facebook API" or "Twitter API" and sends the official account information to the device.

[1188] 6. The official account information received by the device is displayed within the news application.

[1189] 7. The user clicks the Follow button from the displayed list of official accounts.

[1190] 8. The device sends a request to the server that the follow button has been pressed.

[1191] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[1192] 10. The server notifies the device of the results of the follow process, and the device displays a message to the user indicating that the follow has been completed.

[1193] This system allows users to quickly and accurately identify the official social media accounts of people featured in news articles and easily follow them.

[1194] Example prompts to input to the generative AI model

[1195] prompt:

[1196] Describe a system that identifies the official social media accounts of celebrities mentioned in news articles. This system analyzes image and video data from news articles and provides users with the social media account information of the identified people, allowing them to easily follow them.

[1197] scenario:

[1198] 1. A user opens an article in a news application that features a famous actor.

[1199] 2. The device retrieves the image and video data from the article and sends it to the server.

[1200] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[1201] 4. The server compares the extracted features with a database to identify famous actors.

[1202] 5. The server uses an API to search for the official social media account for the identified actor's name and sends the official account information to the device.

[1203] 6. The device displays the received official account information in the news application.

[1204] 7. The user presses the follow button from the displayed list of official accounts.

[1205] 8. The device sends a request to the server that the follow button was pressed.

[1206] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[1207] 10. The server notifies the terminal of the results of the follow process, and the terminal displays a message to the user indicating that the follow has been completed.

[1208] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1209] Program processing flow

[1210] Step 1:

[1211] A user launches a news application and browses to a news article of interest.

[1212] Specifically, a user selects an article featuring a famous actor from the application's news feed, opens the article, and begins reading.

[1213] Input: User launches news application and selects article

[1214] Output: Display article content

[1215] Step 2:

[1216] The device acquires image and video data from the news article the user is viewing.

[1217] The device automatically extracts all image and video data contained in the article and temporarily stores it.

[1218] Input: News article data

[1219] Output: Extraction of image and video data

[1220] Step 3:

[1221] The image and video data acquired by the device is sent to the server.

[1222] The terminal collects all the extracted image and video data into packets and transmits them to the server.

[1223] Input: Image and video data

[1224] Output: Send data to the server

[1225] Step 4:

[1226] The image and video data received by the server is analyzed using an AI model (e.g., TensorFlow).

[1227] The server inputs the received data into an AI model and extracts the person's features, using facial recognition technology.

[1228] Input: Image and video data

[1229] Output: Feature extraction

[1230] Step 5:

[1231] The server identifies the person based on the extracted features by comparing them with a database.

[1232] The server compares the extracted features with a database of pre-trained people and identifies matching people.

[1233] Input: Feature data, database

[1234] Output: Identified person

[1235] Step 6:

[1236] The server searches for the official social media accounts of the identified person from various social media platforms (e.g., Facebook API, Twitter API).

[1237] The server uses the API of the social media platform to search for official accounts based on the identified person's name and collects that information.

[1238] Input: Name of the identified person

[1239] Output: Official social media account information

[1240] Step 7:

[1241] The server sends the collected official SNS account information to the device.

[1242] The server assembles the collected account information into packets and sends them to the terminal.

[1243] Input: Official SNS account information

[1244] Output: Send account information to the device

[1245] Step 8:

[1246] The SNS account information received by the device is displayed within the news application.

[1247] The device displays the received official SNS accounts in a list format on the user interface.

[1248] Input: Official SNS account information

[1249] Output: Display account information

[1250] Step 9:

[1251] The user clicks the follow button for the account of interest from the displayed list of official accounts.

[1252] The user selects the desired account from the displayed list and clicks the follow button.

[1253] Input: Official SNS account list

[1254] Output: Follow button click

[1255] Step 10:

[1256] The device sends a request to the server that the follow button has been pressed.

[1257] The terminal sends the user's follow request to the server, and the request includes the user's SNS authentication information.

[1258] Input: Follow request

[1259] Output: Request sent to server

[1260] Step 11:

[1261] The server follows the official account based on the user's social media authentication information.

[1262] The server executes the follow process using the SNS API based on the received request.

[1263] Input: Follow request, social media credentials

[1264] Output: Follow processing result

[1265] Step 12:

[1266] The server notifies the terminal of the follow-up processing result.

[1267] The server notifies the terminal of the success or failure of the follow process.

[1268] Input: Follow-up processing result

[1269] Output: Notification to terminal

[1270] Step 13:

[1271] The device displays a follow-up completion message to the user.

[1272] The terminal displays a message to the user indicating that the follow has been completed based on the result of the follow process received from the server.

[1273] Input: Notification from the server

[1274] Output: Display of follow-up completion message

[1275] (Application example 1)

[1276] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1277] There is a need for a system that can identify the official social media accounts of people appearing in images and videos displayed on web pages and allow users to easily follow them, but with existing technology it has been difficult to automate this process, and users have had to perform cumbersome operations. There is a need to solve these issues and improve the user experience.

[1278] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1279] In this invention, the server includes means for identifying a person appearing in an image or video on a webpage using image recognition technology, means for searching for the official information acquisition service account of the identified person, means for allowing a user to follow the official information acquisition service account within the website, means for displaying the official information acquisition service account information of the identified person on a user interface, and means for automatically following the official information acquisition service account based on the account information acquired by the user. This allows a user to easily follow a person's official SNS account without performing complicated operations.

[1280] "Image recognition technology" is a technology that extracts useful information from video data and identifies people and objects.

[1281] A "web page" is a document containing information that is viewable on the Internet.

[1282] "Images and videos" refers to still images and moving video data that contain visual information.

[1283] "People" refers to people appearing in an image or video.

[1284] An "official information acquisition service account" is an account that is authenticated on a social networking service (SNS).

[1285] "Searching" is the process of locating specific information.

[1286] A "user" is a person who uses the system.

[1287] A "user interface" is a screen and operating means for a user to interact with a system.

[1288] "Following" refers to linking with a specific account on social media and receiving information.

[1289] A "server" is a computer system that processes data on a network.

[1290] The system for implementing this invention consists of three main components: a server, a terminal, and a user.

[1291] 1. Server Processing

[1292] The server receives image and video data from web pages and uses image recognition technology to identify people. Software such as TensorFlow and PIL (Python Imaging Library) is installed on the server, which processes and calculates the received image and video data. Specifically, features are extracted from the image data and a generative AI model is used to match the data with a database to identify the person. The identified person's official social media account is searched for via the API of each social media platform. The search results are saved for subsequent processing.

[1293] 2. Terminal Processing

[1294] The terminal is a device that displays the web page operated by the user, and has the role of displaying the account information of the official information acquisition service received from the server on the user interface. When the user presses the follow button, the request is sent from the terminal to the server.

[1295] 3. User Operation

[1296] While browsing a website, users may want to follow the official information service account of a person who appears in an image or video on the page. In this case, users can easily follow the official account by pressing the follow button displayed on their device.

[1297] Specific examples

[1298] For example, suppose a user is browsing a favorite product page on a fashion shopping site. The product page features an image of a famous model. Using this system, the following happens:

[1299] 1. When a user opens a product page, the device sends the image data of that page to the server.

[1300] 2. The server uses TensorFlow to perform image analysis and extract model features.

[1301] 3. Based on the features, the server matches the database and identifies the model.

[1302] 4. The server searches for the identified model's official information retrieval service account (e.g., Instagram or Twitter) using the API of each social media platform.

[1303] 5. The retrieved account information is sent to the device and displayed on the user's web page.

[1304] 6. Users can follow the model's official account from their own social media account by clicking the follow button that appears.

[1305] Prompt Sentence Examples

[1306] Example prompts to input to generative AI models such as Chat GPT:

[1307] We are developing a system that uses image analysis technology to identify the official social media accounts of people featured in specific news articles and automatically follow those accounts. Please generate a prompt sentence under the following conditions:

[1308] The system first retrieves an image / video of the page the user is viewing.

[1309] The captured images / videos are sent to the server and people are identified using TensorFlow.

[1310] Search for the official social media accounts of the identified person using the social media API.

[1311] Displays SNS account information to users. Users can follow official accounts by clicking the follow button.

[1312] Finally, we display a message to the user confirming that they have followed us.

[1313] This allows a system to be realized that allows users to easily follow a person's official SNS account when browsing a web page.

[1314] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1315] Step 1:

[1316] The device acquires image and video data from the web page the user is viewing. The URL of the web page and a reference image are given as input. The device then sends the acquired image data to the server. In this step, the device collects image data using a browser or a specific application installed on the device, converts the received data into an appropriate format, and passes it to the server.

[1317] Step 2:

[1318] The server processes the image and video data received from the device. First, it uses image recognition technology to extract the features of the people in the image. Image data is given as input, and the person's identification information is obtained as output. Then, TensorFlow is used to perform image analysis and extract specific features (such as facial contours and the placement of the eyes, nose, and mouth). This generates data that can be used to identify specific people in the image or video.

[1319] Step 3:

[1320] The server uses the extracted features to match the database and identify the person. Feature data is given as input, and the identified person's identity is obtained as output. The server uses a generative AI model to quickly search the existing database and return the corresponding person information. This step identifies who the person in the image is.

[1321] Step 4:

[1322] The server searches each SNS platform for the identified person's official information retrieval service account. The person's identification information is given as input, and SNS account information is obtained as output. The server uses the SNS API to send a search query based on the identified person's name and collects official account information. In this step, only authenticated accounts are extracted and listed.

[1323] Step 5:

[1324] The server sends the identified official information retrieval service account information to the device. The social media account information is given as input, and account information displayed in the user interface is generated as output. The server sends formatted data so that the information can be displayed correctly on the device. This data includes the official mark, account name, and profile URL.

[1325] Step 6:

[1326] The device displays the received official information acquisition service account information on a user interface. The account information sent from the server is given as input, and information that can be visually confirmed by the user is displayed as output. The device appropriately arranges the information on a web page and provides interactive elements including a follow button.

[1327] Step 7:

[1328] The user presses the Follow button from the displayed list of official information acquisition service accounts. The user's action is taken as input, and a follow request is generated as output. In this step, the Follow button responds to the user's action as a trigger, and the next action is taken.

[1329] Step 8:

[1330] When the user presses the follow button, the device sends a follow request to the server. User operation information is given as input, and the follow request is sent to the server as output. The device generates follow request data and makes a specific API call to the server.

[1331] Step 9:

[1332] The server receives a follow request from a user and automatically follows a specific official information retrieval service account based on that information. The follow request information is given as input, and a follow completion status is generated as output. The server executes the follow action through the SNS API.

[1333] Step 10:

[1334] The server notifies the device of the results of the follow process, and the device displays a message to the user indicating that the follow has been completed. The follow completion information is given as input, and a confirmation message is generated as output and displayed to the user. The server and device work together to allow the user to confirm that the follow was successful.

[1335] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1336] This invention combines a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles, allowing users to easily follow those accounts, with an emotion engine that recognizes the user's emotions. The program for this system is described in detail below.

[1337] Overall system overview

[1338] The system consists of three main components: a server, a terminal, and a user.

[1339] 1. Server

[1340] The server receives image and video data from the news application and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It also uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow.

[1341] 2. Terminal

[1342] The terminal is a device that runs a news application operated by the user. The terminal displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[1343] 3. Users

[1344] Users operate the news application to browse news articles of interest. They can easily follow the official social media accounts of people featured in images and videos within the article. They can also select more appropriate follow-up actions by receiving suggestions based on the analysis results of the emotion engine.

[1345] Program processing

[1346] 1. Image and video analysis

[1347] When a user operates a news application and opens a news article, the device retrieves the images and video data from the article and sends them to the server. The server then inputs the received image and video data into an AI model and extracts the features of the people appearing in the images and videos.

[1348] 2. Identifying people and searching social media accounts

[1349] The server identifies the person displayed by checking the extracted features against an internal database. It then searches for the identified person's official social media accounts via the APIs of various social media platforms, identifies accounts with the official account mark (verification badge), and extracts this as official account information.

[1350] 3. User Emotion Recognition and Analysis

[1351] The device captures the user's facial expressions, voice, operation patterns, etc. in real time and sends this data to the server. The server uses an emotion engine to analyze this data and recognize the user's emotional state. Emotional information indicates the user's basic emotional state, such as joy, anger, sadness, and happiness.

[1352] 4. Providing account information and sentiment analysis results

[1353] The server creates suggestions for following SNS accounts that match the user's emotions based on the official SNS account information and the results of the user's emotion analysis, and sends the suggestions to the device. The device then displays the received information on the interface within the news application.

[1354] 5. Implementing the Follow Feature

[1355] The user presses the follow button from the list of official accounts displayed and suggestions based on the sentiment analysis results. When the follow button is pressed, the device sends a follow request to the server. This request also includes the user's SNS authentication information.

[1356] 6. Execute follow-up process

[1357] The server receives the follow request and performs authentication procedures on each SNS platform based on the SNS authentication information. After successful authentication, the API is used to follow the target official account from the user's account. The server notifies the device of the follow processing results, and the device displays a message to the user indicating that the follow has been completed.

[1358] Specific examples

[1359] Scenario: A user views a news article featuring a famous actor.

[1360] 1. A user opens an article in a news application that mentions a famous actor.

[1361] 2. The device retrieves the image and video data from the article and sends it to the server.

[1362] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[1363] 4. The server compares the features with the database to identify famous actors.

[1364] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[1365] 6. The device acquires the user's facial expressions, voice, and operation patterns and sends them to the server.

[1366] 7. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the device.

[1367] 8. The device displays the received information within the app.

[1368] 9. The user presses the Follow button from the displayed list of accounts and suggestions.

[1369] 10. The device sends the follow button click information to the server.

[1370] 11. The server performs authentication procedures on each SNS based on the SNS authentication information and completes the follow process using the API.

[1371] 12. The server sends the results of the follow process to the terminal, and the terminal displays a message to the user indicating that the follow has been completed.

[1372] This system allows users to easily follow the official social media accounts of people featured in news articles based on sentiment-based suggestions.

[1373] The processing flow will be explained below.

[1374] Step 1:

[1375] A user navigates through a news application and opens a news article of interest.

[1376] Step 2:

[1377] The device acquires image and video data from the article and sends that data to the server.

[1378] Step 3:

[1379] The server inputs the received image and video data into the AI ​​model and extracts the features of the people appearing in the images and videos.

[1380] Step 4:

[1381] The server compares the extracted features with an internal database to identify the person being displayed.

[1382] Step 5:

[1383] Based on the information of the identified person, the server uses the APIs of various social media platforms to search for that person's official social media accounts.

[1384] Step 6:

[1385] The server identifies accounts that have official account marks (e.g., verification badges) and extracts this as official account information.

[1386] Step 7:

[1387] The terminal acquires data such as the user's facial expressions, voice, and operation patterns, and sends this data to the server.

[1388] Step 8:

[1389] The server inputs the received data into the emotion engine and analyzes the user's emotional state. For example, if the user smiles, it identifies the emotional state as "joy," and if the user frowns, it identifies the emotional state as "dissatisfaction."

[1390] Step 9:

[1391] Based on the results of the emotion analysis, the server creates suggestions for following official social media accounts that match the user's emotions.

[1392] Step 10:

[1393] The server sends official SNS account information and emotion analysis results to the device.

[1394] Step 11:

[1395] The device will display the received official social media account information and follow suggestions on the interface within the news application.

[1396] Step 12:

[1397] Users can click the follow button from the displayed list of official accounts and suggestions based on sentiment analysis results.

[1398] Step 13:

[1399] When the user clicks the follow button, the device sends a follow request to the server, including the user's SNS authentication information.

[1400] Step 14:

[1401] The server receives the follow request and performs authentication procedures on each SNS platform based on the user's SNS authentication information.

[1402] Step 15:

[1403] After successful authentication, the server uses the API to follow the target official account from the user's account.

[1404] Step 16:

[1405] The server notifies the terminal of the result of the follow-up process.

[1406] Step 17:

[1407] The device notifies the user by displaying a message in the news application indicating that the follow has been completed.

[1408] Example 2

[1409] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1410] In conventional news applications, when users want to follow the official social networking service (SNS) accounts of people featured in news articles, they must manually search for them, which is time-consuming. Furthermore, there is insufficient information to determine whether the account is of interest to the user, and simply providing account information does not sufficiently stimulate the user's interest. This not only requires time and effort to follow, but also results in a lack of improvement in the user experience.

[1411] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1412] In this invention, the server includes means for identifying people appearing in images and videos of news articles using image recognition technology, means for searching for the identified people's official social networking service (SNS) accounts, means for allowing users to follow the official SNS accounts within the news application, means for acquiring the user's emotional data in real time, means for analyzing the user's emotional state using an emotion engine, and means for creating follow suggestions based on the analyzed emotional information. This allows users to easily follow the official SNS accounts of people appearing in news articles, and further provides a more engaging experience by receiving appropriate follow suggestions based on emotions.

[1413] "Image recognition technology" is a technology that identifies and analyzes specific features and patterns from digital images and videos.

[1414] A "news article" refers to the informational content published in the media, including multimedia content such as text, images, and video.

[1415] "Person identification" is the process of identifying people in images or videos using specific methods and algorithms.

[1416] A "social networking service (SNS)" is an online platform that enables users to communicate with each other over the Internet.

[1417] An "official SNS account" is an account officially authenticated by an individual or organization on a social networking service.

[1418] "Following" is the act of registering another user on a social networking site with the aim of receiving their posts and information on a regular basis.

[1419] "Emotion data" is data that indicates the user's emotional state, and is obtained from facial expressions, voice, operation patterns, and the like.

[1420] An "emotion engine" is software or an algorithm that analyzes a user's emotional state based on collected emotional data.

[1421] "Follow suggestions" are suggestions that recommend social media accounts for users to follow based on analyzed information and the user's interests.

[1422] A "server" is a computer system that processes data and provides information in response to user requests.

[1423] A "terminal" is a device that allows a user to operate a news application, and includes a smartphone, tablet, or the like.

[1424] This invention is a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles, allowing users to easily follow those accounts, and combines this with an emotion engine that recognizes the user's emotions. This system consists of three main components: a server, a terminal, and a user.

[1425] Overall system overview

[1426] 1. Server

[1427] The server receives image and video data from the news application and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It then uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow.

[1428] 2. Terminal

[1429] The terminal is a device that runs a news application operated by the user. The terminal displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[1430] 3. Users

[1431] Users operate the news application to browse news articles of interest. They can easily follow the official social media accounts of people featured in images and videos within the article. They can also select more appropriate follow-up actions by receiving suggestions based on the analysis results of the emotion engine.

[1432] Hardware and software used

[1433] The server is a server machine equipped with a high-performance processor that runs an AI model using Python (e.g., TensorFlow, OpenCV). The emotion engine uses an emotion analysis service such as Microsoft Azure Emotion API. The terminal is a device such as a smartphone or tablet that exchanges data with the server in real time using WebSocket technology. The terminal is also equipped with a camera and microphone and has the ability to capture the user's facial expressions and voice.

[1434] Specific examples

[1435] Scenario: A user views a news article featuring a famous actor.

[1436] 1. A user opens an article in a news application that mentions a famous actor.

[1437] 2. The device retrieves the image and video data from the article and sends it to the server.

[1438] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[1439] 4. The server compares the features with the database to identify famous actors.

[1440] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[1441] 6. The device acquires the user's facial expressions, voice, and operation patterns and sends them to the server.

[1442] 7. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the device.

[1443] 8. The device displays the received information within the app.

[1444] 9. The user presses the Follow button from the displayed list of accounts and suggestions.

[1445] 10. The device sends the follow button click information to the server.

[1446] 11. The server performs authentication procedures on each SNS based on the SNS authentication information and completes the follow process using the API.

[1447] 12. The server sends the results of the follow process to the terminal, and the terminal displays a message to the user indicating that the follow has been completed.

[1448] Prompt Sentence Examples

[1449] "Please explain how you input image data into your AI model and extract features of people in the image. Please be specific about the process, including the specific libraries and tools you use."

[1450] This system allows users to easily follow the official social media accounts of people featured in news articles based on sentiment-based suggestions.

[1451] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1452] Step 1:

[1453] A user launches a news application and selects a particular news article, which includes images and video.

[1454] Input: User action to select a news article

[1455] Output: Start retrieving image and video data from articles

[1456] What happens: A user opens a news application on their smartphone and clicks on an article of interest from the news feed.

[1457] Step 2:

[1458] The device acquires image and video data from the article, and sends the acquired data to the server.

[1459] Input: URL of image or video in news article

[1460] Output: Binary image and video data

[1461] Specific operation: The device extracts the URLs of images and videos from the DOM of the article displayed by the news application, and sends the obtained image and video data in binary format to the server.

[1462] Step 3:

[1463] The server inputs the received image and video data into the AI ​​model and extracts the features of the people in the images.

[1464] Input: binary image or video data

[1465] Output: Extracted person feature data

[1466] How it works: The server uses Python libraries (e.g., TensorFlow, OpenCV) to input image data into the AI ​​model, and the model extracts facial features (eye position, nose shape, jaw line, etc.).

[1467] Step 4:

[1468] The server compares the extracted features with an internal database to identify the person being displayed.

[1469] Input: Extracted person feature data

[1470] Output: Data of identified person

[1471] Specific operation: The server compares the extracted feature data with an internal database and obtains the ID and name of the identified person.

[1472] Step 5:

[1473] The server searches for the identified person's official social media accounts using the APIs of various social media platforms, extracts official accounts (with verification badges) from the search results, and sends that information to the device.

[1474] Input: ID or name of the identified person

[1475] Output: Official social media account information (with verification badge)

[1476] Specific operation: The server uses the Twitter API or Facebook Graph API to search for official social media accounts using the identified person's name, extracts official account information with a verification badge from the obtained account data, and sends it to the device in JSON format.

[1477] Step 6:

[1478] The device acquires emotional data such as the user's facial expressions, voice, and operation patterns in real time and transmits this data to the server.

[1479] Input: User's facial expression, voice, and operation pattern data

[1480] Output: Collected emotion data

[1481] Specific operation: The device uses a camera and microphone to capture the user's facial expressions and voice, and also records operation data such as the user's touch pattern and scrolling speed, and sends it to the server via WebSocket.

[1482] Step 7:

[1483] The server uses an emotion engine to analyze the received emotion data and recognize the user's emotional state. Based on the results, it creates suggestions for following SNS accounts suitable for the user and sends them to the device.

[1484] Input: Collected emotion data

[1485] Output: Sentiment analysis results and follow suggestion data

[1486] How it works: The server uses an emotion analysis library (e.g., Microsoft Azure Emotion API) to analyze facial expressions and voice data to identify the user's emotional state (joy, anger, sadness, etc.), and then generates follow suggestions based on this and sends the information to the device.

[1487] Step 8:

[1488] The device displays the received SNS account information and emotion analysis results on the user interface and sends a request to the server for the user to press the follow button.

[1489] Input: Follow suggestion data

[1490] Output: User's follow button operation information

[1491] Specific operation: The device displays official social media account information and sentiment analysis results on the interface within the news app, receives instructions from the user to press the follow button for accounts that interest them, and sends this information to the server.

[1492] Step 9:

[1493] The server receives the click information of the follow button, performs authentication procedures with the corresponding SNS based on the SNS authentication information, and then completes the follow process using the API.

[1494] Input: User follow button operation information, SNS authentication information

[1495] Output: Follow processing result

[1496] Specific operation: The server uses the SNS API to send a follow request based on the user's SNS authentication information and receives a response indicating whether the follow process was successful.

[1497] Step 10:

[1498] The server notifies the terminal of the result of the follow process, and the terminal displays a message indicating that the follow has been completed to the user.

[1499] Input: Follow-up processing result data

[1500] Output: Follow completion message

[1501] Specific operation: The server sends data to the device indicating that the follow process has been completed, and the device then pops up a "Follow Complete" message to the user based on that data.

[1502] This series of processes allows users to easily follow the official social media accounts of people featured in news articles, and by receiving appropriate follow suggestions based on their emotions, users can have a more engaging experience.

[1503] (Application example 2)

[1504] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1505] Conventional news applications have the drawback of requiring users to identify and follow the social media accounts of people featured in articles, and of not suggesting appropriate accounts to follow that reflect the user's interests. Furthermore, in the advertising field, there is a lack of functionality that allows users to easily identify the social media accounts of people featured in advertisements and follow them based on their emotions. Therefore, there is a need for a method to increase user engagement and maximize advertising effectiveness.

[1506] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1507] In this invention, the server includes means for identifying people appearing in images and videos of news articles using image recognition technology, means for searching for the official social networking service (SNS) accounts of the identified people, means for analyzing the user's emotions using an emotion engine that recognizes the user's emotions, means for suggesting appropriate official SNS accounts to follow based on the analyzed emotions, and means for identifying people in advertisements based on image and video analysis of the advertisements and searching for their SNS accounts. This enables users to easily follow the official SNS accounts of people appearing in news articles and advertisements based on emotion-based suggestions.

[1508] "Image recognition technology" is a technology for identifying and analyzing people and objects from images and video data acquired using cameras and sensors.

[1509] A "news article" is a document or content that describes news reports published in newspapers, websites, etc.

[1510] A "video" is a medium that expresses movement by playing back a series of multiple images along a time axis.

[1511] "Person" refers to an individual human being who appears in a news article or advertisement.

[1512] A "social networking service (SNS)" is an online platform that allows users to share information and communicate.

[1513] An "account" is user identification information required to use an SNS.

[1514] "Following" is the act of setting up a social networking site to receive updates from a specific user.

[1515] An "emotion engine" is a software technology for analyzing emotions from a user's facial expressions, voice, operation patterns, etc.

[1516] "Analysis" is the process of examining data in detail and understanding its structure and meaning.

[1517] "Follow Suggestions" are recommendations of social media accounts to follow that are displayed to users based on analyzed sentiment and other data.

[1518] "Advertisements" are promotional content such as images and videos created to advertise products or services.

[1519] "Image and video analysis" is the process of analyzing images and videos within an advertisement to identify people and objects contained within them.

[1520] This invention combines a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos in news articles and advertisements, allowing users to easily follow those accounts, with an emotion engine that recognizes the user's emotions. The details of the program for this system are described below.

[1521] Overall system overview

[1522] The system consists of three main components: a server, a terminal, and a user.

[1523] 1. Server:

[1524] The server receives image and video data from news applications and advertising videos and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It then uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow. The server uses OpenCV for image collection and analysis, and TensorFlow and Keras for machine learning models. It also uses Azure Face API and Google Cloud Vision API for emotion analysis.

[1525] 2. Terminal:

[1526] The device is a device on which the news application and advertisement display application operated by the user runs. The device displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, it sends the request to the server. The device must be equipped with a camera and microphone.

[1527] 3. User:

[1528] Users operate a news application or an advertising display application to view news articles or advertisements that interest them. They can easily follow the official social media accounts of people featured in images or videos in the articles or advertisements. They can also receive suggestions based on the analysis results of the emotion engine, allowing them to select more appropriate follow-up actions.

[1529] Specific examples

[1530] Scenario: User viewing an advertisement for a fashion brand

[1531] 1. A user views a fashion brand advertisement on their smartphone. At this time, the smartphone's camera captures the images and video data in the advertisement and sends them to a server.

[1532] 2. The server analyzes the received data using OpenCV and TensorFlow and extracts the features of the model appearing in the advertisement.

[1533] 3. The server uses Elasticsearch to match the features with the database to identify the person appearing in the advertisement. It then uses various social media APIs (e.g., Twitter API, Instagram Graph API) to send the identified model's official social media account information to the smartphone.

[1534] 4. The smartphone captures the user's facial expressions, voice, and operation patterns in real time and sends them to the server using the Azure Face API, Google Cloud Vision API, and Google Cloud Speech-to-Text API.

[1535] 5. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the smartphone.

[1536] 6. The smartphone displays the received information on the user interface, overlaid on the advertisement, and the user presses the follow button from the displayed account list and suggestions.

[1537] 7. The smartphone sends the click information of the follow button to the server. The server performs authentication procedures with each SNS based on the SNS authentication information and completes the follow process using the API.

[1538] 8. The server sends the follow-up processing results to the smartphone, and the smartphone displays a completion message to the user.

[1539] Prompt Sentence Examples

[1540] Design an application that allows users to easily follow the official social media accounts of models appearing in fashion brand advertisements while viewing them. The application will analyze images and videos from the advertisement to identify the models and search for their social media accounts. It will also analyze users' sentiment in real time and make follow suggestions based on that sentiment. Explain the necessary hardware and software, as well as the specific steps involved.

[1541] In this way, users can easily follow the official social media accounts of people featured in news articles and advertisements based on emotion-based suggestions.

[1542] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1543] Step 1:

[1544] The user operates a news application or an advertisement display application on the device to view news articles or advertisements that interest them. The device acquires the displayed image or video data and sends it to the server.

[1545] Input: Image and video data of news articles and advertisements

[1546] Output: Send data to the server

[1547] How it works: Your smartphone's camera captures images and video data and sends it over the internet to a server.

[1548] Step 2:

[1549] The server analyzes the received image and video data using OpenCV and TensorFlow, and extracts the features of the people appearing in the images and videos.

[1550] Input: Image and video data

[1551] Output: Person features

[1552] How it works: The server analyzes the captured image and video data, inputs it into an AI model, and extracts the person's features.

[1553] Step 3:

[1554] The server uses Elasticsearch to match the extracted features with an internal database to identify the person in the photo. It then searches for the identified person's official social media accounts via the APIs of various social media platforms (e.g., Twitter API, Instagram Graph API) and extracts official account information.

[1555] Input: Person features

[1556] Output: Official SNS account information

[1557] How it works: The server compares the features with an internal database and searches for social media accounts based on the identified person's name.

[1558] Step 4:

[1559] The device captures the user's facial expressions, voice, and operation patterns in real time and sends this data to the server. The device's camera and microphone capture emotional data to identify the user's emotions.

[1560] Input: User's facial expressions, voice, and operation patterns

[1561] Output: Send emotion data to the server

[1562] How it works: The device captures emotional data in real time using a camera and microphone and sends it to a server.

[1563] Step 5:

[1564] The server uses an emotion engine (e.g., Azure Face API, Google Cloud Vision API, Google Cloud Speech-to-Text API) to analyze the user's emotions, and based on the results, creates follow suggestions and sends them to the device.

[1565] Input: Emotion data

[1566] Output: Follow suggestion information

[1567] How it works: The server uses the emotion engine to analyze the emotion data and generates follow-up suggestions based on the analysis results.

[1568] Step 6:

[1569] The device displays the received official SNS account information and sentiment analysis results on the user interface of the news application or advertisement display application, allowing the user to press the follow button.

[1570] Input: Follow suggestion information

[1571] Output: Display in the user interface

[1572] How it works: The device updates the user interface with the follow suggestion information and displays a follow button.

[1573] Step 7:

[1574] When a user presses the follow button, the device sends a follow request to the server, which also includes the user's social media authentication information.

[1575] Input: Follow button click information, SNS authentication information

[1576] Output: Send follow request to server

[1577] Operation: The device sends the follow button click information and SNS authentication information to the server.

[1578] Step 8:

[1579] The server receives the follow request and performs authentication procedures on each SNS platform based on the SNS authentication information. After successful authentication, the server uses the API to follow the target official account from the user's account. The device is notified of the follow processing results.

[1580] Input: Follow request, SNS authentication information

[1581] Output: Follow processing results

[1582] Operation: The server performs authentication procedures on each SNS platform and executes the follow process.

[1583] Step 9:

[1584] The terminal displays the follow-up processing result received from the server to the user as a completion message.

[1585] Input: Follow-up processing result

[1586] Output: Display of completion message

[1587] Operation: The device displays a completion message on the screen based on the results of the follow-up process.

[1588] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1589] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1590] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1591] [Fourth embodiment]

[1592] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1593] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1594] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1595] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1596] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1597] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1598] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1599] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1600] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1601] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1602] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1603] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1604] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1605] This invention provides a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles and allows users to easily follow those accounts. The program for this system is described in detail below.

[1606] Overall system overview

[1607] The system consists of three main components: a server, a terminal, and a user.

[1608] 1. Server

[1609] The server receives image and video data from the news application, identifies people using image recognition technology, searches for the identified people's official social media accounts on various social media platforms, and sends that information to the device.

[1610] 2. Terminal

[1611] The device is a device on which a news application operated by a user runs. The device displays the official SNS account information received from the server on a user interface, and when the user presses the follow button, the device sends the request to the server.

[1612] 3. Users

[1613] Users operate the news application to view news articles they are interested in. They can easily follow the official social media accounts of people featured in images and videos within the articles.

[1614] Program processing

[1615] 1. Image and video analysis

[1616] The device acquires image and video data from the news article the user is viewing and sends it to a server. The server then uses image recognition technology to identify people appearing in the images and videos based on the received data. The server then uses the latest AI models to extract features and compare them with a database to identify the person.

[1617] 2. Search for official social media accounts

[1618] After the person is identified, the server searches for the identified person's social media accounts on various social media platforms. Specifically, the server collects the official social media accounts via API based on the identified person's name and stores them in a database.

[1619] 3. Provide the user with account information

[1620] The server sends the search results to the device, which then displays the official social media account information within the news application, allowing users to view the official accounts of their interest on the news application interface.

[1621] 4. Implementing the Follow Feature

[1622] When a user presses the follow button from the displayed list of official accounts, the device sends the request to the server. The server automatically follows the official account based on the user's SNS authentication information. The server notifies the device of the completion of the follow and displays it to the user.

[1623] Specific examples

[1624] Scenario: A user views a news article featuring a famous actor.

[1625] 1. A user opens an article in a news application that features a famous actor.

[1626] 2. The device retrieves the image and video data from the article and sends it to the server.

[1627] 3. The server analyzes the received data using an "AI model" and extracts the actor's characteristics.

[1628] 4. The server compares the extracted features with a database to identify famous actors.

[1629] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[1630] 6. The device displays the received official account information in the news application.

[1631] 7. The user presses the follow button from the displayed list of official accounts.

[1632] 8. The device sends a request to the server that the follow button was pressed.

[1633] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[1634] 10. The server notifies the terminal of the results of the follow process, and the terminal displays a message to the user indicating that the follow has been completed.

[1635] This will create a system that allows users to easily follow the official social media accounts of people featured in news articles.

[1636] The processing flow will be explained below.

[1637] Step 1:

[1638] A user navigates through a news application and opens a news article of interest.

[1639] Step 2:

[1640] The device extracts image and video data contained in the news article and sends that data to the server.

[1641] Step 3:

[1642] The server inputs the received image and video data into the AI ​​model and extracts the features of the people appearing in the images and videos.

[1643] Step 4:

[1644] The server compares the extracted features with an internal database to identify the person being displayed.

[1645] Step 5:

[1646] Based on the information of the identified person, the server uses the APIs of various social media platforms to search for that person's official social media accounts.

[1647] Step 6:

[1648] The server identifies accounts that have official account marks (e.g., verification badges) and extracts this as official account information.

[1649] Step 7:

[1650] The server sends official SNS account information to the device.

[1651] Step 8:

[1652] The device displays the received official SNS account information on the interface within the news application.

[1653] Step 9:

[1654] The user clicks the follow button from the displayed list of official accounts.

[1655] Step 10:

[1656] When the user clicks the follow button, the device sends a follow request to the server, including the user's SNS authentication information.

[1657] Step 11:

[1658] The server receives the follow request and performs authentication procedures on each SNS platform based on the user's SNS authentication information.

[1659] Step 12:

[1660] After the server successfully authenticates the user, it uses the API to follow the target official account from the user's account.

[1661] Step 13:

[1662] The server notifies the terminal of the result of the follow-up process.

[1663] Step 14:

[1664] The device notifies the user by displaying a message in the news application indicating that the follow has been completed.

[1665] This series of processes allows users to easily follow the official social media accounts of people featured in news articles.

[1666] Example 1

[1667] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1668] Conventional news applications lack a means to quickly and accurately identify the official social networking service (SNS) accounts of people featured in articles and allow users to easily follow them. Furthermore, analyzing images and videos takes time, making it difficult to provide accurate account information.

[1669] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1670] In this invention, the server includes a means for a terminal to acquire and transmit image and video data of a news article, a means for the server to analyze the received image and video data and identify a person using image recognition technology, a means for the server to search for the identified person's official social networking service (SNS) account from various SNS platforms, a means for the server to perform a follow process based on the user's SNS authentication information, and a means for the terminal to display official SNS account information to the user and provide a follow button, thereby enabling the user to quickly and accurately identify the official SNS account of a person appearing in a news article and easily follow the person.

[1671] A "terminal" is a device on which a user operates a news application, and is generally an electronic device such as a smartphone, tablet, or PC.

[1672] "Images and video data of news articles" refers to visual content contained in articles provided within the news application, including still images and video data.

[1673] "Server" refers to a central processing unit that processes data received from a terminal and transmits analysis results and additional information to the terminal.

[1674] "Image recognition technology" refers to technology that uses computer vision techniques to identify people and objects in images and videos.

[1675] An "SNS platform" is an online system that provides social networking services, and generally refers to networks such as Twitter and Facebook.

[1676] An "AI model" refers to mathematical algorithms and data structures created based on artificial intelligence technology to automatically perform specific tasks (such as person identification).

[1677] "Features" refer to important attributes and patterns extracted from images and videos, and are used to identify people and objects.

[1678] "Official social media account" refers to an official social media account that is verified on a social media platform, such as the official profile of a celebrity or company.

[1679] A "follow button" is a button that is displayed on the user interface, and a user can click on it to follow a specific SNS account.

[1680] MODE FOR CARRYING OUT THE INVENTION

[1681] This invention provides a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles and enables users to easily follow those accounts. Detailed embodiments of this system are described below.

[1682] System Configuration

[1683] This system mainly consists of a server, terminals, and users.

[1684] 1. Server

[1685] The server receives image and video data sent from the news application and uses image recognition technology to identify people.

[1686] The server searches for the identified person's official social media account on various social media platforms and sends that information to the device.

[1687] The server is equipped with AI models such as TensorFlow and OpenCV, which are used to analyze the received data.

[1688] 2. Terminal

[1689] The terminal is a device on which a news application operated by a user runs, and includes a smartphone, tablet, PC, etc.

[1690] The terminal displays the official SNS account information received from the server on the user interface, and when the user presses the follow button, sends the request to the server.

[1691] 3. Users

[1692] The user operates the news application to view news articles of interest.

[1693] Users can easily follow the official social media accounts of people featured in images and videos within articles using the follow button.

[1694] Operation overview

[1695] Data Acquisition and Transmission

[1696] When a user uses a news application to view a news article, the device acquires the image and video data contained in the article and transmits it to a server.

[1697] Analysis and Identification

[1698] The server uses the latest AI models to extract features from the received image and video data, and compares them with a database to identify people, using image recognition technologies such as TensorFlow and OpenCV.

[1699] Account Search

[1700] The server searches for official accounts using the APIs of various social media platforms based on the identified person's name. The collected social media account information is temporarily stored in a database.

[1701] Information provision

[1702] The server sends the search results to the terminal, and the terminal displays the official SNS account information on the user interface.

[1703] Follow-up process

[1704] When a user presses the follow button, the device sends the request to the server. The server automatically follows the official account based on the user's SNS authentication information. The device is notified of the success or failure of this process, and displays a message to the user indicating that the follow has been completed.

[1705] Specific examples

[1706] Scenario: A user views a news article featuring a famous actor.

[1707] 1. A user opens an article in a news application that mentions a famous actor.

[1708] 2. The device retrieves the image and video data from the article and sends it to the server.

[1709] 3. The server analyzes the received data using TensorFlow and extracts the actor's features.

[1710] 4. The server compares the extracted features with a database to identify famous actors.

[1711] 5. The server searches for the official social media account using the identified actor's name using the "Facebook API" or "Twitter API" and sends the official account information to the device.

[1712] 6. The official account information received by the device is displayed within the news application.

[1713] 7. The user clicks the Follow button from the displayed list of official accounts.

[1714] 8. The device sends a request to the server that the follow button has been pressed.

[1715] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[1716] 10. The server notifies the device of the results of the follow process, and the device displays a message to the user indicating that the follow has been completed.

[1717] This system allows users to quickly and accurately identify the official social media accounts of people featured in news articles and easily follow them.

[1718] Example prompts to input to the generative AI model

[1719] prompt:

[1720] Describe a system that identifies the official social media accounts of celebrities mentioned in news articles. This system analyzes image and video data from news articles and provides users with the social media account information of the identified people, allowing them to easily follow them.

[1721] scenario:

[1722] 1. A user opens an article in a news application that features a famous actor.

[1723] 2. The device retrieves the image and video data from the article and sends it to the server.

[1724] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[1725] 4. The server compares the extracted features with a database to identify famous actors.

[1726] 5. The server uses an API to search for the official social media account for the identified actor's name and sends the official account information to the device.

[1727] 6. The device displays the received official account information in the news application.

[1728] 7. The user presses the follow button from the displayed list of official accounts.

[1729] 8. The device sends a request to the server that the follow button was pressed.

[1730] 9. The server follows the official account via the SNS API based on the user's SNS authentication information.

[1731] 10. The server notifies the terminal of the results of the follow process, and the terminal displays a message to the user indicating that the follow has been completed.

[1732] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1733] Program processing flow

[1734] Step 1:

[1735] A user launches a news application and browses to a news article of interest.

[1736] Specifically, a user selects an article featuring a famous actor from the application's news feed, opens the article, and begins reading.

[1737] Input: User launches news application and selects article

[1738] Output: Display article content

[1739] Step 2:

[1740] The device acquires image and video data from the news article the user is viewing.

[1741] The device automatically extracts all image and video data contained in the article and temporarily stores it.

[1742] Input: News article data

[1743] Output: Extraction of image and video data

[1744] Step 3:

[1745] The image and video data acquired by the device is sent to the server.

[1746] The terminal collects all the extracted image and video data into packets and transmits them to the server.

[1747] Input: Image and video data

[1748] Output: Send data to the server

[1749] Step 4:

[1750] The image and video data received by the server is analyzed using an AI model (e.g., TensorFlow).

[1751] The server inputs the received data into an AI model and extracts the person's features, using facial recognition technology.

[1752] Input: Image and video data

[1753] Output: Feature extraction

[1754] Step 5:

[1755] The server identifies the person based on the extracted features by comparing them with a database.

[1756] The server compares the extracted features with a database of pre-trained people and identifies matching people.

[1757] Input: feature data, database

[1758] Output: Identified person

[1759] Step 6:

[1760] The server searches for the official social media accounts of the identified person from various social media platforms (e.g., Facebook API, Twitter API).

[1761] The server uses the API of the social media platform to search for official accounts based on the identified person's name and collects that information.

[1762] Input: Name of the identified person

[1763] Output: Official social media account information

[1764] Step 7:

[1765] The server sends the collected official SNS account information to the device.

[1766] The server assembles the collected account information into packets and sends them to the terminal.

[1767] Input: Official SNS account information

[1768] Output: Send account information to the device

[1769] Step 8:

[1770] The SNS account information received by the device is displayed within the news application.

[1771] The device displays the received official SNS accounts in a list format on the user interface.

[1772] Input: Official SNS account information

[1773] Output: Display account information

[1774] Step 9:

[1775] The user clicks the follow button for the account of interest from the displayed list of official accounts.

[1776] The user selects the desired account from the displayed list and clicks the follow button.

[1777] Input: Official SNS account list

[1778] Output: Follow button click

[1779] Step 10:

[1780] The device sends a request to the server that the follow button has been pressed.

[1781] The terminal sends the user's follow request to the server, and the request includes the user's SNS authentication information.

[1782] Input: Follow request

[1783] Output: Request sent to the server

[1784] Step 11:

[1785] The server follows the official account based on the user's social media authentication information.

[1786] The server executes the follow process using the SNS API based on the received request.

[1787] Input: Follow request, social media credentials

[1788] Output: Follow processing result

[1789] Step 12:

[1790] The server notifies the terminal of the follow-up processing result.

[1791] The server notifies the terminal of the success or failure of the follow process.

[1792] Input: Follow-up processing result

[1793] Output: Notification to terminal

[1794] Step 13:

[1795] The device displays a follow-up completion message to the user.

[1796] The terminal displays a message to the user indicating that the follow has been completed based on the result of the follow process received from the server.

[1797] Input: Notification from the server

[1798] Output: Display of follow-up completion message

[1799] (Application example 1)

[1800] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1801] There is a need for a system that can identify the official social media accounts of people appearing in images and videos displayed on web pages and allow users to easily follow them, but with existing technology it has been difficult to automate this process, and users have had to perform cumbersome operations. There is a need to solve these issues and improve the user experience.

[1802] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1803] In this invention, the server includes means for identifying a person appearing in an image or video on a webpage using image recognition technology, means for searching for the official information acquisition service account of the identified person, means for allowing a user to follow the official information acquisition service account within the website, means for displaying the official information acquisition service account information of the identified person on a user interface, and means for automatically following the official information acquisition service account based on the account information acquired by the user. This allows a user to easily follow a person's official SNS account without performing complicated operations.

[1804] "Image recognition technology" is a technology that extracts useful information from video data and identifies people and objects.

[1805] A "web page" is a document containing information that is viewable on the Internet.

[1806] "Images and videos" refers to still images and moving video data that contain visual information.

[1807] "People" refers to people appearing in an image or video.

[1808] An "official information acquisition service account" is an account that is authenticated on a social networking service (SNS).

[1809] "Searching" is the process of locating specific information.

[1810] A "user" is a person who uses the system.

[1811] A "user interface" is a screen and operating means for a user to interact with a system.

[1812] "Following" refers to linking with a specific account on social media and receiving information.

[1813] A "server" is a computer system that processes data on a network.

[1814] The system for implementing this invention consists of three main components: a server, a terminal, and a user.

[1815] 1. Server Processing

[1816] The server receives image and video data from web pages and uses image recognition technology to identify people. Software such as TensorFlow and PIL (Python Imaging Library) is installed on the server, which processes and calculates the received image and video data. Specifically, features are extracted from the image data and a generative AI model is used to match the data with a database to identify the person. The identified person's official social media account is searched for via the API of each social media platform. The search results are saved for subsequent processing.

[1817] 2. Terminal Processing

[1818] The terminal is a device that displays the web page operated by the user, and has the role of displaying the account information of the official information acquisition service received from the server on the user interface. When the user presses the follow button, the request is sent from the terminal to the server.

[1819] 3. User Operation

[1820] While browsing a website, users may want to follow the official information service account of a person who appears in an image or video on the page. In this case, users can easily follow the official account by pressing the follow button displayed on their device.

[1821] Specific examples

[1822] For example, suppose a user is browsing a favorite product page on a fashion shopping site. The product page features an image of a famous model. Using this system, the following happens:

[1823] 1. When a user opens a product page, the device sends the image data of that page to the server.

[1824] 2. The server uses TensorFlow to perform image analysis and extract model features.

[1825] 3. Based on the features, the server matches the database and identifies the model.

[1826] 4. The server searches for the identified model's official information acquisition service account (e.g., Instagram or Twitter) using the API of each social media platform.

[1827] 5. The retrieved account information is sent to the device and displayed on the user's web page.

[1828] 6. Users can follow the model's official account from their own social media account by clicking the follow button that appears.

[1829] Prompt Sentence Examples

[1830] Example prompts to input to generative AI models such as Chat GPT:

[1831] We are developing a system that uses image analysis technology to identify the official social media accounts of people featured in a particular news article and automatically follow those accounts. Please generate a prompt sentence under the following conditions:

[1832] The system first retrieves an image / video of the page the user is viewing.

[1833] The captured images / videos are sent to the server and people are identified using TensorFlow.

[1834] Search for the official social media accounts of the identified person using the social media API.

[1835] Displays SNS account information to users. Users can follow official accounts by clicking the follow button.

[1836] Finally, we display a message to the user confirming that they have followed us.

[1837] This allows a system to be realized that allows users to easily follow a person's official SNS account when browsing a web page.

[1838] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1839] Step 1:

[1840] The device acquires image and video data from the web page the user is viewing. The URL of the web page and a reference image are given as input. The device then sends the acquired image data to the server. In this step, the device collects image data using a browser or a specific application installed on the device, converts the received data into an appropriate format, and passes it to the server.

[1841] Step 2:

[1842] The server processes the image and video data received from the device. First, it uses image recognition technology to extract the features of the people in the image. Image data is given as input, and the person's identification information is obtained as output. Then, TensorFlow is used to perform image analysis and extract specific features (such as facial contours and the placement of the eyes, nose, and mouth). This generates data that can be used to identify specific people in the image or video.

[1843] Step 3:

[1844] The server uses the extracted features to match the database and identify the person. Feature data is given as input, and the identified person's identity is obtained as output. The server uses a generative AI model to quickly search the existing database and return the corresponding person information. This step identifies who the person in the image is.

[1845] Step 4:

[1846] The server searches each SNS platform for the identified person's official information retrieval service account. The person's identification information is given as input, and SNS account information is obtained as output. The server uses the SNS API to send a search query based on the identified person's name and collects official account information. In this step, only authenticated accounts are extracted and listed.

[1847] Step 5:

[1848] The server sends the identified official information retrieval service account information to the device. The social media account information is given as input, and account information displayed in the user interface is generated as output. The server sends formatted data so that the information can be displayed correctly on the device. This data includes the official mark, account name, and profile URL.

[1849] Step 6:

[1850] The device displays the received official information acquisition service account information on a user interface. The account information sent from the server is given as input, and information that can be visually confirmed by the user is displayed as output. The device appropriately arranges the information on a web page and provides interactive elements including a follow button.

[1851] Step 7:

[1852] The user presses the Follow button from the displayed list of official information acquisition service accounts. The user's action is taken as input, and a Follow request is generated as output. In this step, the Follow button responds to the user's action as a trigger, and the next action is taken.

[1853] Step 8:

[1854] When the user presses the follow button, the device sends a follow request to the server. User operation information is given as input, and the follow request is sent to the server as output. The device generates follow request data and makes a specific API call to the server.

[1855] Step 9:

[1856] The server receives a follow request from a user and automatically follows a specific official information retrieval service account based on that information. The follow request information is given as input, and a follow completion status is generated as output. The server executes the follow action through the SNS API.

[1857] Step 10:

[1858] The server notifies the device of the results of the follow process, and the device displays a message to the user indicating that the follow has been completed. The follow completion information is given as input, and a confirmation message is generated as output and displayed to the user. The server and device work together to allow the user to confirm that the follow was successful.

[1859] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1860] This invention combines a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles, allowing users to easily follow those accounts, with an emotion engine that recognizes the user's emotions. The program for this system is described in detail below.

[1861] Overall system overview

[1862] The system consists of three main components: a server, a terminal, and a user.

[1863] 1. Server

[1864] The server receives image and video data from the news application and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It also uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow.

[1865] 2. Terminal

[1866] The terminal is a device that runs a news application operated by the user. The terminal displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[1867] 3. Users

[1868] Users operate the news application to browse news articles of interest. They can easily follow the official social media accounts of people featured in images and videos within the article. They can also select more appropriate follow-up actions by receiving suggestions based on the analysis results of the emotion engine.

[1869] Program processing

[1870] 1. Image and video analysis

[1871] When a user operates a news application and opens a news article, the device retrieves the images and video data from the article and sends them to the server. The server then inputs the received image and video data into an AI model and extracts the features of the people appearing in the images and videos.

[1872] 2. Identifying people and searching social media accounts

[1873] The server identifies the person displayed by checking the extracted features against an internal database. It then searches for the identified person's official social media accounts via the APIs of various social media platforms, identifies accounts with the official account mark (verification badge), and extracts this as official account information.

[1874] 3. User Emotion Recognition and Analysis

[1875] The device captures the user's facial expressions, voice, operation patterns, etc. in real time and sends this data to the server. The server uses an emotion engine to analyze this data and recognize the user's emotional state. Emotional information indicates the user's basic emotional state, such as joy, anger, sadness, and happiness.

[1876] 4. Providing account information and sentiment analysis results

[1877] The server creates suggestions for following SNS accounts that match the user's emotions based on the official SNS account information and the results of the user's emotion analysis, and sends the suggestions to the device. The device then displays the received information on the interface within the news application.

[1878] 5. Implementing the Follow Feature

[1879] The user presses the follow button from the list of official accounts displayed and suggestions based on the sentiment analysis results. When the follow button is pressed, the device sends a follow request to the server. This request also includes the user's SNS authentication information.

[1880] 6. Execute follow-up process

[1881] The server receives the follow request and performs authentication procedures on each SNS platform based on the SNS authentication information. After successful authentication, the API is used to follow the target official account from the user's account. The server notifies the device of the follow processing results, and the device displays a message to the user indicating that the follow has been completed.

[1882] Specific examples

[1883] Scenario: A user views a news article featuring a famous actor.

[1884] 1. A user opens an article in a news application that mentions a famous actor.

[1885] 2. The device retrieves the image and video data from the article and sends it to the server.

[1886] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[1887] 4. The server compares the features with the database to identify famous actors.

[1888] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[1889] 6. The device acquires the user's facial expressions, voice, and operation patterns and sends them to the server.

[1890] 7. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the device.

[1891] 8. The device displays the received information within the app.

[1892] 9. The user presses the Follow button from the displayed list of accounts and suggestions.

[1893] 10. The device sends the follow button click information to the server.

[1894] 11. The server performs authentication procedures on each SNS based on the SNS authentication information and completes the follow process using the API.

[1895] 12. The server sends the results of the follow process to the terminal, and the terminal displays a message to the user indicating that the follow has been completed.

[1896] This system allows users to easily follow the official social media accounts of people featured in news articles based on sentiment-based suggestions.

[1897] The processing flow will be explained below.

[1898] Step 1:

[1899] A user navigates through a news application and opens a news article of interest.

[1900] Step 2:

[1901] The device acquires image and video data from the article and sends that data to the server.

[1902] Step 3:

[1903] The server inputs the received image and video data into the AI ​​model and extracts the features of the people appearing in the images and videos.

[1904] Step 4:

[1905] The server compares the extracted features with an internal database to identify the person being displayed.

[1906] Step 5:

[1907] Based on the information of the identified person, the server uses the APIs of various social media platforms to search for that person's official social media accounts.

[1908] Step 6:

[1909] The server identifies accounts that have official account marks (e.g., verification badges) and extracts this as official account information.

[1910] Step 7:

[1911] The terminal acquires data such as the user's facial expressions, voice, and operation patterns, and sends this data to the server.

[1912] Step 8:

[1913] The server inputs the received data into the emotion engine and analyzes the user's emotional state. For example, if the user smiles, it identifies the emotional state as "joy," and if the user frowns, it identifies the emotional state as "dissatisfaction."

[1914] Step 9:

[1915] Based on the results of the emotion analysis, the server creates suggestions for following official social media accounts that match the user's emotions.

[1916] Step 10:

[1917] The server sends official SNS account information and emotion analysis results to the device.

[1918] Step 11:

[1919] The device will display the received official social media account information and follow suggestions on the interface within the news application.

[1920] Step 12:

[1921] Users can click the follow button from the displayed list of official accounts and suggestions based on sentiment analysis results.

[1922] Step 13:

[1923] When the user clicks the follow button, the device sends a follow request to the server, including the user's SNS authentication information.

[1924] Step 14:

[1925] The server receives the follow request and performs authentication procedures on each SNS platform based on the user's SNS authentication information.

[1926] Step 15:

[1927] After successful authentication, the server uses the API to follow the target official account from the user's account.

[1928] Step 16:

[1929] The server notifies the terminal of the result of the follow-up process.

[1930] Step 17:

[1931] The device notifies the user by displaying a message in the news application indicating that the follow has been completed.

[1932] Example 2

[1933] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1934] In conventional news applications, when users want to follow the official social networking service (SNS) accounts of people featured in news articles, they must manually search for them, which is time-consuming. Furthermore, there is insufficient information to determine whether the account is of interest to the user, and simply providing account information does not sufficiently stimulate the user's interest. This not only requires time and effort to follow, but also results in a lack of improvement in the user experience.

[1935] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1936] In this invention, the server includes means for identifying people appearing in images and videos of news articles using image recognition technology, means for searching for the identified people's official social networking service (SNS) accounts, means for allowing users to follow the official SNS accounts within the news application, means for acquiring the user's emotional data in real time, means for analyzing the user's emotional state using an emotion engine, and means for creating follow suggestions based on the analyzed emotional information. This allows users to easily follow the official SNS accounts of people appearing in news articles, and further provides a more engaging experience by receiving appropriate follow suggestions based on emotions.

[1937] "Image recognition technology" is a technology that identifies and analyzes specific features and patterns from digital images and videos.

[1938] A "news article" refers to the informational content published in the media, including multimedia content such as text, images, and video.

[1939] "Person identification" is the process of identifying people in images or videos using specific methods and algorithms.

[1940] A "social networking service (SNS)" is an online platform that enables users to communicate with each other over the Internet.

[1941] An "official SNS account" is an account officially authenticated by an individual or organization on a social networking service.

[1942] "Following" is the act of registering another user on a social networking site with the aim of receiving their posts and information on a regular basis.

[1943] "Emotion data" is data that indicates the user's emotional state, and is obtained from facial expressions, voice, operation patterns, and the like.

[1944] An "emotion engine" is software or an algorithm that analyzes a user's emotional state based on collected emotional data.

[1945] "Follow suggestions" are suggestions that recommend social media accounts for users to follow based on analyzed information and the user's interests.

[1946] A "server" is a computer system that processes data and provides information in response to user requests.

[1947] A "terminal" is a device that allows a user to operate a news application, and includes a smartphone, tablet, or the like.

[1948] This invention is a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos displayed in news articles, allowing users to easily follow those accounts, and combines this with an emotion engine that recognizes the user's emotions. This system consists of three main components: a server, a terminal, and a user.

[1949] Overall system overview

[1950] 1. Server

[1951] The server receives image and video data from the news application and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It then uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow.

[1952] 2. Terminal

[1953] The terminal is a device that runs a news application operated by the user. The terminal displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, the terminal sends the request to the server.

[1954] 3. Users

[1955] Users operate the news application to browse news articles of interest. They can easily follow the official social media accounts of people featured in images and videos within the article. They can also select more appropriate follow-up actions by receiving suggestions based on the analysis results of the emotion engine.

[1956] Hardware and software used

[1957] The server is a server machine equipped with a high-performance processor that runs an AI model using Python (e.g., TensorFlow, OpenCV). The emotion engine uses an emotion analysis service such as Microsoft Azure Emotion API. The terminal is a device such as a smartphone or tablet that exchanges data with the server in real time using WebSocket technology. The terminal is also equipped with a camera and microphone and has the ability to capture the user's facial expressions and voice.

[1958] Specific examples

[1959] Scenario: A user views a news article featuring a famous actor.

[1960] 1. A user opens an article in a news application that mentions a famous actor.

[1961] 2. The device retrieves the image and video data from the article and sends it to the server.

[1962] 3. The server analyzes the received data using an AI model and extracts the actor's features.

[1963] 4. The server compares the features with the database to identify famous actors.

[1964] 5. The server searches for the official social media account using the identified actor's name and sends the official account information to the device.

[1965] 6. The device acquires the user's facial expressions, voice, and operation patterns and sends them to the server.

[1966] 7. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the device.

[1967] 8. The device displays the received information within the app.

[1968] 9. The user presses the Follow button from the displayed list of accounts and suggestions.

[1969] 10. The device sends the follow button click information to the server.

[1970] 11. The server performs authentication procedures on each SNS based on the SNS authentication information and completes the follow process using the API.

[1971] 12. The server sends the results of the follow process to the terminal, and the terminal displays a message to the user indicating that the follow has been completed.

[1972] Prompt Sentence Examples

[1973] "Please explain how you input image data into your AI model and extract features of people in the image. Please be specific about the process, including the specific libraries and tools you use."

[1974] This system allows users to easily follow the official social media accounts of people featured in news articles based on sentiment-based suggestions.

[1975] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1976] Step 1:

[1977] A user launches a news application and selects a particular news article, which includes images and video.

[1978] Input: User action to select a news article

[1979] Output: Start retrieving image and video data from articles

[1980] What happens: A user opens a news application on their smartphone and clicks on an article of interest from the news feed.

[1981] Step 2:

[1982] The device acquires image and video data from the article, and sends the acquired data to the server.

[1983] Input: URL of image or video in news article

[1984] Output: Binary image and video data

[1985] Specific operation: The device extracts the URLs of images and videos from the DOM of the article displayed by the news application, and sends the obtained image and video data in binary format to the server.

[1986] Step 3:

[1987] The server inputs the received image and video data into the AI ​​model and extracts the features of the people in the images.

[1988] Input: binary image or video data

[1989] Output: Extracted person feature data

[1990] How it works: The server uses Python libraries (e.g., TensorFlow, OpenCV) to input image data into the AI ​​model, and the model extracts facial features (eye position, nose shape, jaw line, etc.).

[1991] Step 4:

[1992] The server compares the extracted features with an internal database to identify the person being displayed.

[1993] Input: Extracted person feature data

[1994] Output: Data of identified person

[1995] Specific operation: The server compares the extracted feature data with an internal database and obtains the ID and name of the identified person.

[1996] Step 5:

[1997] The server searches for the identified person's official social media accounts using the APIs of various social media platforms, extracts official accounts (with verification badges) from the search results, and sends that information to the device.

[1998] Input: ID or name of the identified person

[1999] Output: Official social media account information (with verification badge)

[2000] Specific operation: The server uses the Twitter API or Facebook Graph API to search for official social media accounts using the identified person's name, extracts official account information with a verification badge from the obtained account data, and sends it to the device in JSON format.

[2001] Step 6:

[2002] The device acquires emotional data such as the user's facial expressions, voice, and operation patterns in real time and transmits this data to the server.

[2003] Input: User's facial expression, voice, and operation pattern data

[2004] Output: Collected emotion data

[2005] Specific operation: The device uses a camera and microphone to capture the user's facial expressions and voice, and also records operation data such as the user's touch pattern and scrolling speed, and sends it to the server via WebSocket.

[2006] Step 7:

[2007] The server uses an emotion engine to analyze the received emotion data and recognize the user's emotional state. Based on the results, it creates suggestions for following SNS accounts suitable for the user and sends them to the device.

[2008] Input: Collected emotion data

[2009] Output: Sentiment analysis results and follow suggestion data

[2010] How it works: The server uses an emotion analysis library (e.g., Microsoft Azure Emotion API) to analyze facial expressions and voice data to identify the user's emotional state (joy, anger, sadness, etc.), and then generates follow suggestions based on this and sends the information to the device.

[2011] Step 8:

[2012] The device displays the received SNS account information and emotion analysis results on the user interface and sends a request to the server for the user to press the follow button.

[2013] Input: Follow suggestion data

[2014] Output: User's follow button operation information

[2015] Specific operation: The device displays official social media account information and sentiment analysis results on the interface within the news app, receives instructions from the user to press the follow button for accounts that interest them, and sends this information to the server.

[2016] Step 9:

[2017] The server receives the click information of the follow button, performs authentication procedures with the corresponding SNS based on the SNS authentication information, and then completes the follow process using the API.

[2018] Input: User follow button operation information, SNS authentication information

[2019] Output: Follow processing result

[2020] Specific operation: The server uses the SNS API to send a follow request based on the user's SNS authentication information and receives a response indicating whether the follow process was successful.

[2021] Step 10:

[2022] The server notifies the terminal of the result of the follow process, and the terminal displays a message indicating that the follow has been completed to the user.

[2023] Input: Follow-up processing result data

[2024] Output: Follow completion message

[2025] Specific operation: The server sends data to the device indicating that the follow process has been completed, and the device then pops up a "Follow Complete" message to the user based on that data.

[2026] This series of processes allows users to easily follow the official social media accounts of people featured in news articles, and by receiving appropriate follow suggestions based on their emotions, users can have a more engaging experience.

[2027] (Application example 2)

[2028] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2029] Conventional news applications have the drawback of requiring users to identify and follow the social media accounts of people featured in articles, and of not suggesting appropriate accounts to follow that reflect the user's interests. Furthermore, in the advertising field, there is a lack of functionality that allows users to easily identify the social media accounts of people featured in advertisements and follow them based on their emotions. Therefore, there is a need for a method to increase user engagement and maximize advertising effectiveness.

[2030] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2031] In this invention, the server includes means for identifying people appearing in images and videos of news articles using image recognition technology, means for searching for the official social networking service (SNS) accounts of the identified people, means for analyzing the user's emotions using an emotion engine that recognizes the user's emotions, means for suggesting appropriate official SNS accounts to follow based on the analyzed emotions, and means for identifying people in advertisements based on image and video analysis of the advertisements and searching for their SNS accounts. This enables users to easily follow the official SNS accounts of people appearing in news articles and advertisements based on emotion-based suggestions.

[2032] "Image recognition technology" is a technology for identifying and analyzing people and objects from images and video data acquired using cameras and sensors.

[2033] A "news article" is a document or content that describes news reports published in newspapers, websites, etc.

[2034] A "video" is a medium that expresses movement by playing back a series of multiple images along a time axis.

[2035] "Person" refers to an individual human being who appears in a news article or advertisement.

[2036] A "social networking service (SNS)" is an online platform that allows users to share information and communicate.

[2037] An "account" is user identification information required to use an SNS.

[2038] "Following" is the act of setting up a social networking site to receive updates from a specific user.

[2039] An "emotion engine" is a software technology for analyzing emotions from a user's facial expressions, voice, operation patterns, etc.

[2040] "Analysis" is the process of examining data in detail and understanding its structure and meaning.

[2041] "Follow Suggestions" are recommendations of social media accounts to follow that are displayed to users based on analyzed sentiment and other data.

[2042] "Advertisements" are promotional content such as images and videos created to advertise products or services.

[2043] "Image and video analysis" is the process of analyzing images and videos within an advertisement to identify people and objects contained within them.

[2044] This invention combines a system that identifies the official social networking service (SNS) accounts of people appearing in images and videos in news articles and advertisements, allowing users to easily follow those accounts, with an emotion engine that recognizes the user's emotions. The details of the program for this system are described below.

[2045] Overall system overview

[2046] The system consists of three main components: a server, a terminal, and a user.

[2047] 1. Server:

[2048] The server receives image and video data from news applications and advertising videos and uses image recognition technology to identify people. It then searches for the identified people's official social media accounts on various social media platforms and sends that information to the device. It then uses an emotion engine to analyze the user's emotions and, based on that data, suggests appropriate official social media accounts to follow. The server uses OpenCV for image collection and analysis, and TensorFlow and Keras for machine learning models. It also uses Azure Face API and Google Cloud Vision API for emotion analysis.

[2049] 2. Terminal:

[2050] The device is a device on which the news application and advertisement display application operated by the user runs. The device displays the official SNS account information and sentiment analysis results received from the server on the user interface, and when the user presses the follow button, it sends the request to the server. The device must be equipped with a camera and microphone.

[2051] 3. User:

[2052] Users operate a news application or an advertising display application to view news articles or advertisements that interest them. They can easily follow the official social media accounts of people featured in images or videos in the articles or advertisements. They can also receive suggestions based on the analysis results of the emotion engine, allowing them to select more appropriate follow-up actions.

[2053] Specific examples

[2054] Scenario: User viewing an advertisement for a fashion brand

[2055] 1. A user views a fashion brand advertisement on their smartphone. At this time, the smartphone's camera captures the images and video data in the advertisement and sends them to a server.

[2056] 2. The server analyzes the received data using OpenCV and TensorFlow and extracts the features of the model appearing in the advertisement.

[2057] 3. The server uses Elasticsearch to match the features with the database to identify the person appearing in the advertisement. It then uses various social media APIs (e.g., Twitter API, Instagram Graph API) to send the identified model's official social media account information to the smartphone.

[2058] 4. The smartphone captures the user's facial expressions, voice, and operation patterns in real time and sends them to the server using the Azure Face API, Google Cloud Vision API, and Google Cloud Speech-to-Text API.

[2059] 5. The server uses an emotion engine to analyze the user's emotions, creates follow-up suggestions based on the results, and sends them to the smartphone.

[2060] 6. The smartphone displays the received information on the user interface, overlaid on the advertisement, and the user presses the follow button from the displayed account list and suggestions.

[2061] 7. The smartphone sends the click information of the follow button to the server. The server performs authentication procedures with each SNS based on the SNS authentication information and completes the follow process using the API.

[2062] 8. The server sends the follow-up processing results to the smartphone, and the smartphone displays a completion message to the user.

[2063] Prompt Sentence Examples

[2064] Design an application that allows users to easily follow the official social media accounts of models appearing in fashion brand advertisements while viewing them. The application will analyze images and videos from the advertisement to identify the models and search for their social media accounts. It will also analyze users' sentiment in real time and make follow suggestions based on that sentiment. Explain the necessary hardware and software, as well as the specific steps involved.

[2065] In this way, users can easily follow the official social media accounts of people featured in news articles and advertisements based on emotion-based suggestions.

[2066] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2067] Step 1:

[2068] The user operates a news application or an advertisement display application on the device to view news articles or advertisements that interest them. The device acquires the displayed image or video data and sends it to the server.

[2069] Input: Image and video data of news articles and advertisements

[2070] Output: Send data to the server

[2071] How it works: Your smartphone's camera captures images and video data and sends it over the internet to a server.

[2072] Step 2:

[2073] The server analyzes the received image and video data using OpenCV and TensorFlow, and extracts the features of the people appearing in the images and videos.

[2074] Input: Image and video data

[2075] Output: Person features

[2076] How it works: The server analyzes the captured image and video data, inputs it into an AI model, and extracts the person's features.

[2077] Step 3:

[2078] The server uses Elasticsearch to match the extracted features with an internal database to identify the person in the photo. It then searches for the identified person's official social media accounts via the APIs of various social media platforms (e.g., Twitter API, Instagram Graph API) and extracts official account information.

[2079] Input: Person features

[2080] Output: Official SNS account information

[2081] How it works: The server compares the features with an internal database and searches for social media accounts based on the identified person's name.

[2082] Step 4:

[2083] The device captures the user's facial expressions, voice, and operation patterns in real time and sends this data to the server. The device's camera and microphone capture emotional data to identify the user's emotions.

[2084] Input: User's facial expressions, voice, and operation patterns

[2085] Output: Send emotion data to the server

[2086] How it works: The device captures emotional data in real time using a camera and microphone and sends it to a server.

[2087] Step 5:

[2088] The server uses an emotion engine (e.g., Azure Face API, Google Cloud Vision API, Google Cloud Speech-to-Text API) to analyze the user's emotions, and based on the results, creates follow suggestions and sends them to the device.

[2089] Input: Emotion data

[2090] Output: Follow suggestion information

[2091] How it works: The server uses the emotion engine to analyze the emotion data and generates follow-up suggestions based on the analysis results.

[2092] Step 6:

[2093] The device displays the received official SNS account information and sentiment analysis results on the user interface of the news application or advertisement display application, allowing the user to press the follow button.

[2094] Input: Follow suggestion information

[2095] Output: Display in the user interface

[2096] How it works: The device updates the user interface with the follow suggestion information and displays a follow button.

[2097] Step 7:

[2098] When a user presses the follow button, the device sends a follow request to the server, which also includes the user's social media authentication information.

[2099] Input: Follow button click information, SNS authentication information

[2100] Output: Send follow request to server

[2101] Operation: The device sends the follow button click information and SNS authentication information to the server.

[2102] Step 8:

[2103] The server receives the follow request and performs authentication procedures on each SNS platform based on the SNS authentication information. After successful authentication, the server uses the API to follow the target official account from the user's account. The device is notified of the follow processing results.

[2104] Input: Follow request, SNS authentication information

[2105] Output: Follow processing results

[2106] Operation: The server performs authentication procedures on each SNS platform and executes the follow process.

[2107] Step 9:

[2108] The terminal displays the follow-up processing result received from the server to the user as a completion message.

[2109] Input: Follow-up processing result

[2110] Output: Display of completion message

[2111] Operation: The device displays a completion message on the screen based on the results of the follow-up process.

[2112] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2113] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2114] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2115] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2116] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2117] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2118] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2119] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2120] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way aga...

Claims

1. A method for using image recognition technology to identify people appearing in images and videos from news articles; A means of searching for the identified person's official social networking service (SNS) account; A way for users to follow official social media accounts within the news application, A system including:

2. The system according to claim 1, further comprising means for identifying a person based on features extracted from an image or video.

3. The system according to claim 1, further comprising means for extracting accounts bearing the official account mark from the acquired SNS account information.

Images