Information processing device, control method for information processing device, and control program for information processing device

Abstract

To perform accurate person estimation even when advance learning and face recognition are difficult. [Solution] An information processing device according to one embodiment of the present invention includes a video acquisition unit that acquires video including an estimation target from a shooting device, an estimation target extraction unit that detects the estimation target from images that make up the video and generates an estimation target image that is a partial image of the estimation target in the image, an image feature generation unit that generates image features from the estimation target image, a natural language feature generation unit that generates natural language features of the estimation target image, a search unit that searches for similar candidates according to the image features and natural language features of the estimation target image based on a database that records identification information of the estimation target in association with the image features and natural language features of the estimation target, and an estimation unit that estimates identification information of the estimation target based on the image features and natural language features of similar candidates.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a control program for an information processing device. [Background technology]

[0002] In recent years, various services have been realized using mobile communication technologies that are becoming increasingly faster and with ultra-low latency. For example, the applicants have constructed a mobile base station using 5G (fifth generation mobile communication system) and are providing a multi-angle live streaming service that allows users to watch videos of all 18 holes of a golf tournament in real time and on demand by switching between them (for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] SoftBank Corp., "Demonstration experiment of live video streaming using millimeter-wave 5GSA at the women's golf tournament "Earth Mondamin Cup" ~ Deploying mobile base station vehicles within the golf course and building a communications network for business use specialized in live video streaming ~" [online], June 23, 2023, SoftBank Corp., [Retrieved September 15, 2025], Internet<URL: https: / / www.softbank.jp / corp / news / press / sbkk / 2023 / 20230623_01 / > Summary of the Invention [Means for solving the problem]

[0004] An information processing device according to one embodiment of the present invention includes a video acquisition unit that acquires video including an estimation target from a shooting device, an estimation target extraction unit that detects the estimation target from images that make up the video and generates an estimation target image that is a partial image of the estimation target in the image, an image feature generation unit that generates image features from the estimation target image, a natural language feature generation unit that generates natural language features that describe the features of the estimation target image in natural language, a search unit that searches for similar candidates according to the image features and natural language features of the estimation target image generated from the image based on a database that records identification information of the estimation target in association with image features and natural language features related to the estimation target, and an estimation unit that estimates identification information of the estimation target detected from the image based on an overall score calculated by integrating the image features and natural language features of the similar candidates.

[0005] In an information processing device according to one embodiment of the present invention, the search unit searches for a plurality of similar candidates, and the estimation unit may estimate identification information of an estimation target detected from an image based on the similar candidate with the highest overall score calculated by weighting the similarity of image features and the similarity of natural language features at a predetermined ratio for each of the plurality of similar candidates.

[0006] In an information processing device according to one embodiment of the present invention, the video acquisition unit acquires broadcast video including an image with text information in which text information is displayed, and the estimation target extraction unit generates an estimation target image of the estimation target if the image with text information includes the estimation target, and the information processing device may further include a database update unit that updates the database by associating the text information as identification information of the estimation target with the image features generated by the image feature generation unit and the natural language features generated by the natural language feature generation unit for the estimation target image.

[0007] In an information processing device according to one embodiment of the present invention, the estimation target extraction unit may determine whether an estimation target is included in an image with text information by using a multimodal large-scale language model to which images and text can be input.

[0008] In an information processing device according to one embodiment of the present invention, in response to detection of an estimation target from an image constituting a video acquired from an imaging device, a database update unit may update the database by associating identification information of the estimation target estimated by the estimation unit with image features generated by the image feature generation unit and natural language features generated by the natural language feature generation unit for the estimation target.

[0009] In an information processing device according to one embodiment of the present invention, the estimation target extraction unit may use a database as learning data to detect an estimation target from images constituting a video acquired from an imaging device and generate an estimation target image.

[0010] In the information processing device according to one embodiment of the present invention, the image feature generation unit and the natural language feature generation unit may use a database as learning data to generate image features and natural language features, respectively.

[0011] In an information processing device according to one embodiment of the present invention, at least one image capture device is placed in each of a plurality of sections that constitute a predetermined area through which the estimation target moves, and the estimation target moves through the plurality of sections in a predetermined order, and the information capture device further includes a location information acquisition unit that acquires location information of the image capture device and location information of the estimation target, and the estimation unit may estimate identification information of the estimation target based on the location information of the estimation target, location information of the image capture device that captured the video including the estimation target, and information regarding the order in which the estimation target moves through the plurality of sections.

[0012] An information processing device according to one embodiment of the present invention further includes a distribution unit that distributes video including an estimated object and identification information of the estimated object to be displayed together with the video, and a distribution request acquisition unit that acquires a video distribution request from a viewing terminal, and the distribution unit may distribute video from a shooting device for which video distribution has been requested in the video distribution request, and identification information of the estimated object to be displayed together with the video.

[0013] A control method for an information processing device according to one embodiment of the present invention includes the steps of: the information processing device acquiring a video including an estimation target from a shooting device; detecting the estimation target from an image constituting the video and generating an estimation target image that is a partial image of the estimation target in the image; generating image features from the estimation target image; generating natural language features that describe the features of the estimation target image in natural language; searching for similar candidates according to the image features and natural language features of the estimation target image generated from the image based on a database that records identification information of the estimation target in association with image features and natural language features related to the estimation target; and estimating identification information of the estimation target detected from the image based on an overall score calculated by integrating the image features and natural language features of the similar candidates.

[0014] A control program for an information processing device according to one embodiment of the present invention enables the information processing device to perform the following functions: acquire video including an estimation target from a shooting device; detect the estimation target from images constituting the video and generate an estimation target image that is a partial image of the estimation target in the image; generate image features from the estimation target image; generate natural language features that describe the features of the estimation target image in natural language; search for similar candidates according to the image features and natural language features of the estimation target image generated from the image, based on a database that records identification information of the estimation target in association with image features and natural language features related to the estimation target; and estimate identification information of the estimation target detected from the image, based on an overall score calculated by integrating the image features and natural language features of the similar candidates. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 shows an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a sequence between the components of an information processing system according to an embodiment of the present invention. [Figure 3]FIG. 3 is a flowchart showing an example of processing by the information providing server according to one embodiment of the present invention. [Figure 4] FIG. 4 is a schematic diagram illustrating a process for generating and updating a database and learning data according to one embodiment of the present invention. [Figure 5] FIG. 5 is an example of a database according to an embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart showing an example of processing by the information providing server according to one embodiment of the present invention. [Figure 7] FIG. 7 is a schematic diagram illustrating a process for estimating a person and generating a video for distribution according to an embodiment of the present invention. [Figure 8] 8(a) and (b) are examples of the display screen of the viewing terminal. [Figure 9] FIG. 9 is a block diagram showing an example of the configuration of an information providing server according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the invention (also referred to as the present invention) according to the present disclosure will be described in detail with reference to the drawings. Note that the drawings are merely examples, and the present invention is not limited to those shown in the drawings. For example, the illustrated information processing device (information providing server), imaging device, router, communication terminal, viewing terminal, broadcasting server, database server, network and number of players, information processing system configuration, functional block diagram, example screen, player names, sequence, and flowchart are merely examples, and the present invention is not limited to these.

[0017] In this specification, generative AI, also known as generative AI, generative system AI, or generative AI, includes language models (LLM: Large Language Model / SLM: Small Language Model), GPT (Generative Pre-trained Transformer, registered trademark), Gemini (registered trademark), Claude (registered trademark), Llama (registered trademark), and other language models, and refers to artificial intelligence (including AGI: Artificial General Intelligence or ASI: Artificial Superintelligence) that generates content such as text, images, audio, and video using deep learning technologies such as Transformers, self-attention, and autoregressive networks. Augmentation technologies for generative AI include Retrieval-Augmented Generation (RAG), Memory Augmented Generation, Hybrid Search using Vector Databases, Chunking / Chunk Processing, Knowledge Graph Linkage, Entity Link, and frameworks such as AutoGen, AOG, and LangChain. Techniques for improving the performance of generative AI include fine-tuning using RLHF / RLAIF (Reinforcement Learning from Human Feedback / Reinforcement Learning with AI Feedback), PEFT (Parameter-Efficient Fine-Tuning), LoRA (Low-Rank Adaptation), distillation, quantization, weight sharing, continuous learning, federated learning, and in-context learning.In addition, the operating environment for the generative AI can be any environment, including on-premise, cloud, or edge (on-device), and parallel and distributed learning and inference are also possible using GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), NPUs (Neural Processing Units), IPUs (Intelligence Processing Units), ASICs (Application-Specific Integrated Circuits), and FPGAs (Field-Programmable Gate Arrays).

[0018] As used herein, generative AI may encompass AI agents capable of acting autonomously with specific goals or roles. Generative AI may also constitute so-called multi-agent AI, where two or more AI agents cooperate, collaborate, or collaborate. Generative AI may be applied to a wide variety of applications, including call center support, chatbot-based FAQ auto-reply, AI assistants, translation, summarization, and meeting minutes generation, programming support (code generation, debugging), data analysis, systems development, financial and legal document review, research, diagnosis, drug discovery, design optimization, supply chain management, education, games, the metaverse, advertising, creative marketing, e-commerce (electronic commerce) and business transactions (recommendations), threat intelligence, image / video / music generation, robotics, smart factories, smart cities, traffic control, autonomous driving, and IoT device control. In application areas, generative AI can realize various services, businesses, or operations by integrating multiple tasks. Use cases for generative AI include both internal processing used in systems on the backend and chatbots provided to users on the frontend.

[0019] As used herein, "orchestrator AI (also referred to as orchestration AI)" may refer to an AI that oversees the cooperation, collaboration, and coordination among multiple AI agents. The orchestrator AI may function as a command center as necessary, assigning tasks to two or more AI agents, controlling the execution order, managing the progress of dialogue, and integrating the results. This allows two or more AI agents to cooperate, collaborate, and work together in a chained manner, allowing them to solve a variety of tasks and problems as a collaborative team. However, the cooperation, collaboration, and coordination of two or more AI agents can pose challenges, such as risks of hallucination, fakery, bias, discrimination, data leakage, mistransmission, theft, privacy (personal information) protection, and copyright infringement. Therefore, technologies for managing and monitoring (logging and monitoring) AI agents to ensure security, governance, accountability, auditing, reliability, and compliance may be used.

[0020] In the future, the management and operation of AI agents will require infrastructure such as high-performance GPUs and quantum computers, HPC (High Performance Computing) clusters, distributed / parallel processing, load balancers, CDNs (Content Delivery Networks), multi-access edge computing (MEC), and fog computing. AI agents are expected to gain autonomy while maintaining both reliability and economic efficiency (cost), and to develop beyond the singularity to the AGI or ASI level. The present invention encompasses these future technological trends and applicability, and is not limited to the embodiments described herein.

[0021] As described above, the applicant provides a service that installs dedicated viewing equipment at golf courses and broadcasts high-quality, multi-angle live video captured by multiple cameras installed on the course. For this video, there is a need to provide viewers with information about the players in the video, such as their names. Displaying player names in real time for multiple camera footage requires the installation of personnel and equipment equivalent to the number of cameras, resulting in a significant operational burden. The applicant therefore considered building a system that uses AI (Artificial Intelligence) to estimate player names from video footage. However, in golf competitions, players often wear hats and sunglasses and often look down to focus on the golf ball, making identification using facial recognition difficult. Furthermore, because players do not wear identifying bibs or other identifying items and their clothing changes from day to day, it is difficult to prepare in advance the training data necessary to build an AI model for identification.

[0022] In contrast, according to one embodiment of the present invention, image feature information and text information describing the features in natural language are used to estimate player names. This makes it possible to identify players without using facial images. Furthermore, by utilizing broadcast footage, the learning model required for player name estimation is automatically updated in real time. This makes it possible to continuously achieve highly accurate estimation based on the latest information.

[0023] <System configuration> FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present invention. Note that this diagram is merely an example, and the present invention is not limited thereto. The information processing system 600 may be a system that identifies an estimated target from a video including the estimated target. Here, the "estimated target" may include a person, animal, or object included in the video. The information processing system 600 may particularly be a system that identifies an estimated target from a video including the estimated target moving within a predetermined area. Note that the following description will use a female golf player as an example of the estimated target, and will describe the identification of a female golf player moving around a golf course during a golf tournament. However, the estimated target of the present invention is not limited to this. For example, the estimated target may be a male golf player, or may be any other sports player, regardless of gender, such as a tennis player or a cyclist. Furthermore, the estimated target is not limited to a person, but may also be a group of animals. Furthermore, the estimated target may be a combination of these. The present invention is particularly effective for identifying people in situations where there is no identifying information, such as a bib number or uniform number, and facial recognition is difficult.

[0024] The information processing system 600 may include an information providing server (information processing device) 100, a database server 101, a broadcasting server 300, viewing terminals 200 (200a, 200b), an administrator terminal 400, imaging devices (cameras) 40 (40a, 40b, ..., 40x), a router 41, a communication terminal 42, and a network 500.

[0025] The photographing device 40 (40a, 40b, ..., 40x) may be a camera with at least one installed on each golf course within a golf course. Here, "installed" may mean that the camera is fixed within the golf course or that the cameraman carries the camera. Although multiple photographing devices are shown in FIG. 1, the number is not limited to this. Hereinafter, unless otherwise specified, these photographing devices 40 (40a, 40b, ..., 40x) will be simply referred to as "photographing device 40." Furthermore, the imaging device 40 may be a broadcast camera (ENG (Electronic News Gathering) camera) or a smartphone. The imaging device 40 captures video including the female golf player, who is the estimated target, moving around the golf course. The video captured by the imaging device 40 may be transmitted from the router 41 and the communication terminal 42 to the server 100 and the broadcast server 300 via the network 500. Note that the mode of transmission of the video from the imaging device 40 to the server 100 and the broadcast server 300 is not particularly limited. For example, the video may be transmitted from the imaging device 40 to the broadcast server 300 using an optical cable.

[0026] The information providing server 100 may be capable of executing various processes related to the information providing service realized by the information processing system 600. The information providing server 100 may be connected to the viewing terminal 200, the administrator terminal 400, the broadcast server 300, and the image capturing device 40 via a network 500. The network 500 may include a wireless network or a wired network. Specifically, the network 500 may be, for example, a wireless LAN (WLAN), a wide area network (WAN), code division multiple access (CDMA), long term evolution (LTE), LTE-Advanced, fourth generation communication (4G), fifth generation communication (5G), sixth generation communication (6G) or later mobile communication system, etc. The network 500 is not limited to these examples and may generally be, for example, Bluetooth (registered trademark), an optical fiber line, etc. The network 500 may also be a combination of these.

[0027] The information providing server 100 may be configured as a physical hardware device, or may be a software-based virtual server configured using virtualization technology. Furthermore, it may be a distributed server system in which multiple servers cooperate by communicating over a network, a cloud server built in a cloud environment, or an edge server that distributes and executes predetermined processing near terminals. That is, the server 100 may be configured in a physical environment, a virtual environment, a cloud environment, or a combination of these, and may be realized as a single or multiple devices, or as a logical configuration that realizes the functions of these devices.

[0028] Although detailed processing will be described later, in one embodiment of the present invention, the information providing server 100 estimates the names of players included in an image using feature information of an image constituting video captured by the image capture device 40 and text information describing the features in natural language. The information providing server 100 may perform a combination of AI model-based processing, such as detecting people in the image, removing the background, determining whether a person is a player, extracting image features, grouping similar features, describing image features in natural language (extracting natural language features), and calculating similarity with a database based on image features and natural language features. The database stores image features and natural language features of each player in association with the player's name. The information providing server 100 may perform the above processing when a new image of a player is acquired and update the database using the processing results. The updated database may also be used as training data for the AI ​​model.

[0029] The broadcast server 300 receives video images transmitted from multiple camera devices 40 within the golf course. The broadcast server 300 can generate broadcast video images by adding information about the players in the received video images. The information about the players may be, for example, the player's name, score, or current hole number. This information may be added by inputting it by an operator in the broadcast center, or the operator may confirm the addition of information estimated by a predetermined system. However, this information is not necessarily added to the broadcast video images.

[0030] The viewing terminals 200 (200a, 200b) may be installed in, for example, a clubhouse of a golf course, allowing users to view images captured by the imaging device 40. Although two viewing terminals 200 are shown in FIG. 1, the number is not limited to this. Hereinafter, unless otherwise specified, the viewing terminals 200 will be simply referred to as "viewing terminals 200." Although a tablet terminal is shown as the viewing terminal 200 in FIG. 1, the viewing terminal 200 is not limited to this. The viewing terminal 200 may also be, for example, a laptop computer, a smartphone, a desktop computer, or the like. Furthermore, the location where the viewing terminal 200 is installed is not limited to the clubhouse.

[0031] According to one embodiment of the present invention, a predetermined application for viewing video (hereinafter simply referred to as a "viewing app") may be installed in the viewing terminal 200. Then, on the viewing terminal 200, information about the golf player included in the video captured by the camera device 40 may be superimposed on the video. The user may be able to select the camera device 40 that captures the video to be viewed on the viewing terminal 200. Here, the video captured by the camera device 40 may be broadcast video transmitted from the broadcast server 300, but this broadcast video does not necessarily include information such as the player's name and score. In contrast, according to one embodiment of the present invention, the golf player in the video may be estimated, and information about the player may be provided to the user on the viewing terminal 200 along with the broadcast video.

[0032] The administrator terminal 400 is a communication terminal used by an administrator who provides information provision services. The administrator can perform monitoring and maintenance work on the information processing system 600 via the administrator terminal 400. The administrator terminal 400 may transmit requests in response to input operations received from the administrator to the information providing server 100, or may display information received from the information providing server 100 or the broadcasting server 300 on its own terminal.

[0033] The database server 101 may store (contain) various information (data) used in the information provision service. For example, the names of all golf players may be stored in the database server 101 as master information. Furthermore, the database server 101 may store information necessary for progressing a match, such as information regarding the pairings of each group playing on a golf course and the start time. Furthermore, according to one embodiment of the present invention, the database server 101 may store various data used for estimating player names and training an AI model.

[0034] <Embodiment> An embodiment of the present invention will be described with reference to Figures 2 to 8. Figure 2 is a diagram showing an example of a sequence among the camera devices 40 (camera device A 40a, camera device B 40b), the broadcast server 300, the information providing server 100, the database server 101, and the viewing terminal 200 according to an embodiment of the present invention. Note that for simplicity, only two camera devices 40 are shown in the figure, but it should be noted that images from all of the camera devices 40 are used.

[0035] First, a description will be given of sequence S10 relating to the generation and update process of the database and learning data. The broadcast server 300 acquires video from the camera A 40a and the camera B 40b (steps S11 and S12). The broadcast server 300 generates video for broadcast based on the acquired video (step S13). The video for broadcast is provided from the broadcast server 300 to the information providing server 100 (step S14). Next, the information providing server 100 and the database server 101 generate and update the database and learning data (step S15).

[0036] Fig. 3 is a flowchart showing an example of the process of generating and updating the database and learning data (step S15) in the information providing server 100. Fig. 4 is a schematic diagram illustrating the process of generating and updating the database and learning data.

[0037] In FIG. 3, the information providing server 100 acquires broadcast video including a text-information-attached image with text information from the broadcast server 300 (step T11). The text-information-attached image may be, for example, an image with superimposed captions N11 and N12 of the player's identification information (player name), as in images 1 and 2 constituting broadcast video V11 and V12, respectively, in FIG. 4. The information providing server 100 determines whether the text-information-attached image includes a female golf player who is the target of estimation (step T12). In determining whether a player is included, first, a determination may be made as to whether a person is included in the image using, for example, an AI model for object detection. Examples of the object detection AI model that may be used include, but are not limited to, YOLO (You Only Look Once), Detectron2, Faster R-CNN (Faster Region-based Convolutional Neural Network), SSD (Single Shot Multibox Detector), RetinaNet, and DETR (DEtection Transformer).

[0038] If it is determined that a person is included, the server 100 may use a multimodal LLM to determine whether the person is a female golf player, a caddie, or a spectator, which are the subject of estimation. This may be done, for example, by describing general characteristics of female golf players and caddies and prompts for determining whether a person corresponding to the characteristics is included, and inputting the description and the image into the multimodal LLM. If, through these processes, it is determined that the text-information-attached image includes a player (YES in step T12), the information providing server 100 generates an estimation target image, which is a partial image of the estimation target in the text-information-attached image (step T13). That is, as the estimation target image, player image 11 and player image 12 may be cut out from image 1 and image 2, respectively. Note that the cutting out of player images 11 and 12 (removing the background from images 1 and 2) may be performed using an AI model for cutting out. Examples of AI models that may be used for cutting out include, but are not limited to, DeepLabv3+, Mask R-CNN (Mask Region-based Convolutional Neural Network), and BiSeNet (Bilateral Segmentation Network).

[0039] Next, the information providing server 100 generates image features from the estimation target image (step T14). The image features are numerical representations of the color, pattern, hairstyle, and shape of the player's clothing, gloves, and shoes, and may be generated by an AI model such as CLIP. The information providing server 100 also generates natural language features that describe the features of the estimation target image in natural language (step T15). The natural language features are numerical representations of descriptive text such as "red polo shirt and white skirt," "short hair and white hat," and "blue pants and white striped shirt," and may be generated by an AI model such as LLaVA (Large Language and Vision Assistant). The information providing server 100 associates character information as identification information of the estimation target with the image features and natural language features, and updates the database (step T16). The information providing server 100 may acquire the captions N11 and N12 of the images 1 and 2 using, for example, OCR (Optical Character Recognition) and compare them with master information of player names stored in advance in the database server 101. If they are registered as master information, the player images 11 and 12 cut out from the videos V11 and V12 may be stored in the database DB10 in association with the player names. If it is determined in step T12 that the image with text information does not include the estimation target (NO in step T12), or after step T16 is completed, the process may return to step T11 and continue the processing in the above steps.

[0040] Through the above-described processing, image features and natural language features of each player image are stored in database DB10 in association with the player name, which is the identification information of each player. FIG. 4 is a conceptual diagram of database DB10, illustrating that each player name is associated with a clipped image of the player. FIG. 5 shows an example of database DB10. Database DB10 stores names, image features, natural language features, and natural language labels in association with identifiers (IDs) that uniquely identify the target of estimation (players). Here, each feature is vectorized, and non-numeric data such as images and text are mapped uniformly to a numerical space, enabling similarity calculation, identification, search, and other processes. Note that the diagram is an example, and the information stored in database DB10 is not limited to this.

[0041] As described above, according to one embodiment of the present invention, labeled data is automatically generated from an image with text information and the database is updated, which contributes to improving the accuracy of identifying the estimation target.

[0042] The information providing server 100 may use the database DB10 as learning data for each AI model. This allows the AI ​​model to learn the characteristics of each player to be estimated with high accuracy, and also allows the AI ​​model to flexibly update the database when new players are added to the database, thereby improving the discrimination performance and scalability.

[0043] Returning to FIG. 2, the player name estimation process in step S20 will now be described. In one embodiment of the present invention, a user may be able to request video distribution of a desired viewing location from the viewing terminal 200. FIG. 8(a) is an example of a screen on the viewing terminal 200. On screen D10, the locations of the filming devices 40 on the golf course are indicated by camera icons, and by selecting an icon, the user can view, on the viewing terminal 200, video of the golf course filmed by the filming device 40 corresponding to that icon.

[0044] In response to an icon being selected on the screen D10 of the viewing terminal 200, a video distribution request is transmitted from the viewing terminal 200 to the information providing server 100 (step S21 in FIG. 2). The information providing server 100 requests the broadcast server 300 to distribute video for broadcast (step S22). Here, it is assumed that the icon selected by the user corresponds to the camera device A 40a, and the broadcast server 300 acquires video from the camera device A 40a and generates video for broadcast (steps S23 and S24). The information providing server 100 acquires the video for broadcast from the broadcast server 300 (step S25). Next, the information providing server 100 performs a process of estimating people in the video and generating video for distribution (step S26).

[0045] Fig. 6 is a flowchart showing an example of the person estimation / distribution video generation process (step S26) in the information providing server 100. Fig. 7 is a schematic diagram illustrating the person estimation / distribution video generation process. Note that the diagram is an example, and the present invention is not limited to this.

[0046] The information providing server 100 acquires a video including an estimation target moving within a predetermined area (golf course) from a camera device located within the area (step T21). FIG. 7 shows an example of the acquired video. Image 3 constituting video V31 in FIG. 7 does not include a player's name. In response to this, the information providing server 100 detects the estimation target from the images constituting the video and generates an estimation target image, which is a partial image of the estimation target in the image (step T22). That is, the golf player 13, which is the estimation target, may be cut out from image 3 in FIG. 7. The detection of the estimation target and the cutting out of the partial image may be performed using an AI model such as YOLO or DeepLabv3+, as described above. Next, the information providing server 100 generates image features from the estimation target image (step T23). An AI model such as CLIP, as described above, may be used to generate the image features. The information providing server 100 also generates natural language features that describe the features of the estimation target image in natural language (step T24). An AI model such as the LaVA mentioned above may also be used to generate natural language features.

[0047] The information providing server 100 searches for similar candidates according to the image features and natural language features of the estimation target image based on a database storing identification information of the estimation target and the image features and natural language features of the estimation target in association with each other (step T25). This database may be a database generated by the learning data generation and update process (step S15), such as the database DB10 shown in FIG. 7. The database DB10 stores the image features and natural language features of each player image in association with the player name. The information providing server 100 may search for and extract players having features similar to the image features and natural language features of the estimation target image 13 included in the video V31 of FIG. 7 using an approximate nearest neighbor search method such as FAISS (Facebook AI Similarity Search).

[0048] Next, the information providing server 100 estimates the identification information of the estimation target based on a total score calculated by integrating the image features and natural language features of each of the multiple similar candidates extracted by FAISS (step T26).

[0049] This allows the information providing server 100 to select the most appropriate candidate overall by comprehensively evaluating both feature amounts, even for candidates that have high image similarity but low natural language relevance, or vice versa. This allows the information providing server 100 to estimate the identification information of the target with higher accuracy than when relying on a single feature amount.

[0050] For example, if the similarity is 90% for image features and 60% for natural language features, the player name associated with the highest overall score calculated by weighting these similarities at a predetermined ratio may be selected as the player name to be estimated. The predetermined ratio may be, but is not limited to, 70% for image features and 30% for natural language features, and in the above example, the overall score may be 90×0.7+60×0.3=81. In this way, by comparing multiple similar candidates based on their overall scores, the misidentification rate can be reduced and the optimal estimation target can be selected.

[0051] The information providing server 100 may distribute to the viewing terminal 200 a video including the estimation target and identification information of the estimation target to be displayed together with the video. In the example of Fig. 7, videos V41 and V42 in which player names are displayed as captions N41 and N42, respectively, may be distributed to the viewing terminal 200.

[0052] Note that, in response to detection of an estimation target from an image constituting the video acquired from the image capture device 40, the information providing server 100 may update the database DB10 by associating the image feature and natural language feature related to the estimation target with identification information of the estimation target. That is, in the example of Fig. 7, the estimation target image 13 included in image 3 may be associated with the player name in DB10. In this way, the identification information and feature of the estimation target are automatically reflected in the database, which leads to an increase in the amount of learning data used to estimate the estimation target, enabling continuous improvement in identification accuracy and automatic accumulation of learning data.

[0053] FIG. 8(b) shows an example of a more detailed display screen on the viewing terminal 200. Screens D11 and D12 may be example screens that are displayed when camera icons C9 and C10 are selected on screen D10 in FIG. 8(a), respectively. On screens D11 and D12, player names estimated by the above-described process for the players in the broadcast video may be displayed at the bottom of the screen. Furthermore, the right-hand area of ​​each screen may display the player combinations and scores. Note that the display screens of the viewing terminal 200 are not limited to this.

[0054] <Configuration> The hardware configuration and functional configuration of the information providing server 100 will be described with reference to FIG.

[0055] (1) Hardware configuration of the information provider server The information providing server 100 may include a control unit 110, a communication unit 120, and a storage unit 170 as hardware components.

[0056] The control unit 110 is typically a processor and may include a central processing unit (CPU), a microprocessor (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic element (FPGA), etc. The control unit 110 may execute the functions and methods described in each embodiment by reading out a program stored in the storage unit 170 and executing instructions and codes included in the program.

[0057] The storage unit 170 may store various programs and various data required for the operation of the information providing server 100. The storage unit 170 may include, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 170 may also include a memory that provides a working area for the control unit 110.

[0058] The communication unit 120 may be implemented as hardware such as a network interface card (NIC) or a network adapter, communication software, or a combination of these. The communication unit 120 may transmit and receive various data to and from the viewing terminal 200, the broadcast server 300, the image capturing device 40, etc. via the network 500.

[0059] (2) Functional configuration of the information providing server The information providing server 100 may include, as functions realized by the control unit 110, a video acquisition unit 111, an estimation target extraction unit 112, an image feature generation unit 113, a natural language feature generation unit 114, a search unit 115, an estimation unit 116, a distribution unit 117, a database update unit 118, and a location information acquisition unit 119. Of the functional units shown in FIG. 9 , those that are not essential to the above-described embodiments may be omitted. Furthermore, the functions or processes of the functional units may be realized by machine learning or AI (artificial intelligence) to the extent feasible.

[0060] The video acquisition unit 111 acquires video including the estimation target from the imaging device 40. Note that the video acquisition unit 111 may acquire video indirectly from the imaging device 40 by acquiring broadcast video from the broadcast server 300. As described above, the video acquired by the video acquisition unit 111 from the broadcast server 300 may include video with and without text information such as player names.

[0061] The inference target extraction unit 112 detects an inference target from an image constituting the video and generates an inference target image, which is a partial image of the inference target in the image. For example, the inference target extraction unit 112 uses YOLO as an object detection algorithm to detect an inference target present in the image and extracts the partial image to generate an inference target image. In this case, the inference target extraction unit 112 may apply DeepLabv3+ for object contour extraction and region segmentation processing to separate the target region with high accuracy. Furthermore, the inference target extraction unit 112 may determine whether the inference target is included in the image with text information using a multimodal large-scale language model to which images and text can be input. The multimodal large-scale language model may be, for example, LLaVA. This improves the accuracy of determining the inference target included in the image with text information.

[0062] The image feature generating unit 113 generates image feature from the estimation target image. The image feature generating unit 113 may generate high-order image feature from the estimation target image using a convolutional neural network, such as ResNet.

[0063] The natural language feature generation unit 114 generates natural language features that describe the features of the image to be estimated in natural language. The natural language feature generation unit 114 may generate embedding vectors that express the appearance and attributes of the image to be estimated in natural language, i.e., natural language features, using, for example, CLIP or LLaVA.

[0064] The search unit 115 searches for similar candidates according to the image features and natural language features of the estimation target image generated from an image, based on a database that stores identification information of the estimation target in association with the image features and natural language features of the estimation target. The search by the search unit 115 may use FAISS, which is specialized in performing approximate nearest neighbor search on the large-scale vector set described above. When a vector of the image features and a vector of the natural language features of the search target are input into FAISS, the FAISS returns vectors of the top N similar candidates that are closest to the input vectors. At this time, each similar candidate is assigned a score indicating its similarity to the search target.

[0065] The estimation unit 116 estimates the identification information of the target detected from the image based on an overall score calculated by integrating the image features and natural language features of the similar candidates. The estimation unit 116 may estimate the identification information of the target detected from the image based on the similar candidate with the highest overall score calculated by weighting the similarity of the image features and the similarity of the natural language features at a predetermined ratio for each of the multiple similar candidates. In other words, the estimation unit 116 may use the score indicating the similarity returned by FAISS to output the player name associated with the image that is deemed to have the highest similarity among the multiple similar candidates as the estimation result.

[0066] The distribution unit 117 distributes the video including the estimation target and the identification information of the estimation target to be displayed together with the video. That is, the distribution unit 117 may superimpose the player name output by the estimation unit 116 on the video of the player and transmit it to the viewing terminal 200. This makes it possible to provide the video to the viewing terminal 200 in real time with the identification information of the estimation target added.

[0067] The database update unit 118 updates the database by associating text information as identification information of the estimation target with the image features generated by the image feature generation unit 113 and the natural language features generated by the natural language feature generation unit 114 for the estimation target image. The text information may be player names identified from broadcast video with player names distributed from the broadcast server 300. In other words, the database is updated every time the estimation process is performed, thereby improving the accuracy of player name estimation.

[0068] The location information acquisition unit 119 may acquire location information of the camera device 40 and location information of the target to be estimated. In one embodiment of the present invention, at least one camera device 40 may be installed for each of multiple sections (golf courses) that make up a golf course, which is a predetermined area. Furthermore, the player to be estimated moves through the multiple golf courses in a predetermined order. Therefore, the location information acquisition unit 119 may acquire location information for each of the camera devices 40, the camera crew operating the camera device 40, and the player or the caddy or golf bag accompanying the player. Furthermore, since the start time of the competition is determined for each group of players, it is possible to estimate the golf course on which each group of players is located based on the current time. Therefore, the estimation unit 116 may estimate the name of the target to be estimated based on the player's location information, the location information of the camera device 40 that captured the video including the player, and information regarding the order in which the player moved around the golf course. This ensures higher player identification accuracy compared to estimation based on images alone.

[0069] The distribution request acquisition unit 121 acquires a video distribution request from the viewing terminal 200. The distribution unit 117 may distribute the video of the imaging device 40 for which video distribution is requested in the video distribution request, and identification information of the estimated target to be displayed together with the video.

[0070] Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present invention. For example, the functions included in each component, step, etc. can be rearranged so as not to cause logical inconsistencies, and multiple components, steps, etc. can be combined into one or divided, or omitted or modified as appropriate as needed. Furthermore, the configurations shown in the above embodiments may be combined as appropriate.

[0071] For example, in the above description, a person estimation process and a process for generating video for distribution are performed in response to a video distribution request received from the viewing terminal 200, and the database is updated. However, the timing at which person estimation is performed from the video for broadcast and the timing at which the database is updated are not limited to this. The information providing server 100 may continuously receive the video for broadcast from the broadcast server 300, perform person estimation in real time, and update the database accordingly.

[0072] The programs of the embodiments of the present disclosure may be provided in a state stored in a storage medium readable by an information processing device. The storage medium may be capable of storing the programs in a "non-transitory tangible medium." The programs may include, for example, software programs and information processing device programs. When each functional unit of the server 100 is realized by software, the server 100 may function as multiple agents by having the processor execute programs loaded into memory.

[0073] The program of the present disclosure may be implemented using any programming language, such as a scripting language such as JavaScript (registered trademark) or Python, C language, Go language, Swift (registered trademark), Koltin (registered trademark), or Java (registered trademark).

[0074] According to each aspect of the present disclosure described above, by providing network technologies for 5G and beyond, it is possible to contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote inclusive and sustainable industrialization, and build resilient technological infrastructure." [Explanation of symbols]

[0075] 100 Information providing server (information processing device) 110 control section 111 Video acquisition unit 112 Estimation target extraction unit 113 Image feature generation unit 114 Natural Language Feature Generation Unit 115 Search Department 116 Estimation Department 117 Distribution Department 118 Database Update Department 119 Location information acquisition unit 121 Delivery request acquisition unit 120 Communications Department 170 Storage section 101 Database Server 200 viewing devices 300 Broadcast Server 400 Administrator terminal 500 Network 600 Information Processing Systems 40 Imaging equipment 41 Router 42 Communication terminals

Claims

1. an image acquisition unit that acquires an image including an estimation target from an imaging device; an estimation target extraction unit that detects the estimation target from an image that constitutes the video and generates an estimation target image that is a partial image of the estimation target in the image; an image feature amount generation unit that generates an image feature amount from the estimation target image; a natural language feature generation unit that generates natural language features that describe features of the estimation target image in natural language; a search unit that searches for similar candidates according to the image features and natural language features of the estimation target image generated from the image, based on a database that records identification information of the estimation target and image features and natural language features related to the estimation target in association with each other; an estimation unit that estimates identification information of the estimation target detected from the image based on a comprehensive score calculated by integrating image features and natural language features of the similar candidate; An information processing device comprising:

2. the search unit searches for a plurality of similar candidates, the estimation unit estimates identification information of the estimation target detected from the image based on the similar candidate having the highest overall score calculated by weighting the similarity of the image feature amount and the similarity of the natural language feature amount at a predetermined ratio for each of the plurality of similar candidates; The information processing device according to claim 1 .

3. the video acquisition unit acquires a broadcast video including an image with text information on which text information is displayed; the estimation target extraction unit generates an estimation target image of the estimation target when the estimation target is included in the text information-attached image; a database update unit that updates the database by associating the character information with the image feature generated by the image feature generation unit and the natural language feature generated by the natural language feature generation unit for the image to be estimated, as identification information of the estimation target; and The information processing device according to claim 1 , further comprising:

4. the estimation target extraction unit determines whether the estimation target is included in the image with character information by using a multimodal large-scale language model to which images and text can be input; The information processing device according to claim 3 .

5. the database update unit, in response to detection of an estimation target from an image constituting the video acquired from the imaging device, updates the database by associating, for the estimation target, the image feature generated by the image feature generation unit and the natural language feature generated by the natural language feature generation unit with identification information of the estimation target estimated by the estimation unit; The information processing device according to claim 3 .

6. the estimation target extraction unit uses the database as learning data to detect an estimation target from images constituting the video acquired from the imaging device and generate an estimation target image; The information processing device according to claim 3 .

7. the image feature generation unit and the natural language feature generation unit use the database as learning data to generate image features and natural language features, respectively; The information processing device according to claim 3 .

8. At least one of the image capturing devices is disposed for each of a plurality of sections constituting a predetermined area in which the estimation target moves, and the estimation target moves through the plurality of sections in a predetermined order, a position information acquisition unit that acquires position information of the imaging device and position information of the estimation target, the estimation unit estimates identification information of the estimation target based on position information of the estimation target, position information of a camera that captured a video including the estimation target, and information on a movement order of the estimation target through the plurality of sections; The information processing device according to claim 1 .

9. a distribution unit that distributes a video including the estimated target and identification information of the estimated target to be displayed together with the video; Further comprising a distribution request acquisition unit that acquires a video distribution request from the viewing terminal, the distribution unit distributes the image of the image capture device for which distribution of the image is requested in the image distribution request, and identification information of the estimated target to be displayed together with the image. The information processing device according to claim 1 .

10. The information processing device acquiring an image including an estimation target from an imaging device; detecting the estimation target from an image constituting the video, and generating an estimation target image that is a partial image of the estimation target in the image; generating an image feature from the estimation target image; generating natural language features that describe features of the estimation target image in natural language; a step of searching for similar candidates according to the image features and natural language features of the estimation target image generated from the image, based on a database in which identification information of the estimation target and image features and natural language features related to the estimation target are recorded in association with each other; a step of estimating identification information of the estimation target detected from the image based on a comprehensive score calculated by integrating image features and natural language features of the similar candidate; A control method for an information processing device that executes the above.

11. In the information processing device, a function of acquiring an image including the target to be estimated from a photographing device; a function of detecting the estimation target from an image constituting the video and generating an estimation target image that is a partial image of the estimation target in the image; a function of generating image features from the estimation target image; a function of generating natural language features that describe the features of the estimation target image in natural language; a function of searching for similar candidates according to the image features and natural language features of the estimation target image generated from the image, based on a database in which identification information of the estimation target and image features and natural language features related to the estimation target are recorded in association with each other; a function of estimating identification information of the target detected from the image based on a comprehensive score calculated by integrating image features and natural language features of the similar candidate; A control program for an information processing device that realizes the above.

Images