Video semantic labeling method, device and equipment and storage medium

Abstract

The application discloses a video semantic labeling method and device, equipment and storage medium, and belongs to the field of video semantic understanding. The method comprises the following steps: acquiring a video frame sequence in a video; extracting multi-modal information in at least two dimensions from video frames in the video frame sequence; determining a target plot event in at least two plot events based on the combination of the multi-modal information in the at least two dimensions, and determining the plot event of the video frame; and performing semantic labeling on the video frame sequence according to the plot event of the video frame, so as to obtain the semantic information of the video. Without training a related semantic understanding model, the labeling efficiency and accuracy of the semantic information are improved, the user can watch more high-quality video content by pushing the video after semantic labeling to the user, and the video browsing experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video semantic understanding, and in particular to a video semantic labeling method and device, equipment and a storage medium. BACKGROUND

[0002] When a user watches a video, information related to the video is displayed in the video playing interface, such as the video name, video tags (types), highlights, etc.

[0003] A video producer uploads a video to a video platform, and the video platform stores the uploaded video after information labeling. When a user searches for a video through the labeled information, the video platform sends the video corresponding to the labeled information to the user's corresponding client. Taking a game video as an example, a game anchor (video producer) uploads a game video to a video platform, and a machine learning model performs semantic labeling on possible highlights in the game video, outputting semantic information of the game video. For example, the time axis is between 1 minute and 30 seconds to 2 minutes, and the game battle parties perform a team battle, so the video segment is labeled.

[0004] In the above technical solution, a large number of semantic labeling sample videos are needed in the training process of the machine learning model, and the semantic labeling sample videos need to be labeled by artificial means, which increases the difficulty of obtaining a large number of semantic labeling sample videos. When there is a lack of semantic labeling sample videos, the machine learning model trained with fewer semantic labeling sample videos has poor accuracy in semantic labeling. SUMMARY

[0005] The embodiments of the present application provide a video semantic labeling method, device, equipment and storage medium, which determines the matched game event by combining the virtual element information and the game information extracted from the video frames, labels the video according to the game event, obtains the semantic information of the video without the machine learning model trained by the sample video, and improves the labeling efficiency and accuracy of the semantic information. The technical solution is as follows:

[0006] According to an aspect of the present application, a video semantic labeling method is provided, which comprises:

[0007] Obtaining a video frame sequence in a video;

[0008] Extracting multi-modal information of at least two dimensions from the video frames in the video frame sequence;

[0009] Determining a target plot event matched in at least two plot events based on the combination of the multi-modal information of at least two dimensions as the plot event of the video frame;

[0010] According to the plot event of the video frame, the video frame sequence is semantically labeled to obtain semantic information of the video.

[0011] According to another aspect of the present application, a video semantic labeling device is provided, and the device comprises:

[0012] An acquisition module is configured to acquire a video frame sequence in a video.

[0013] An extraction module is configured to extract multi-modal information in at least two dimensions from video frames in the video frame sequence.

[0014] A processing module is configured to determine a plot event of the video frame based on a target plot event matched in at least two plot events based on a combination of the multi-modal information in at least two dimensions.

[0015] A labeling module is configured to label the video frame sequence based on the plot event of the video frame to obtain semantic information of the video.

[0016] According to another aspect of the present application, a computer device is provided, and the computer device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the video semantic labeling method according to the above aspect.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, and the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the video semantic labeling method according to the above aspect.

[0018] According to another aspect of the present application, a computer program product or a computer program is provided, and the computer program product or the computer program comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the video semantic labeling method according to the above aspect.

[0019] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0020] By combining the multi-modal information extracted from the video frame and the matched target plot event, the plot event of the video frame is determined, so that the video frame is semantically annotated by the corresponding plot event of the video frame. The video frame can be annotated with semantic information by a machine learning model trained by a sample video, improving the efficiency and accuracy of semantic annotation of the video frame, so that the user can quickly capture the highlight segment in the video frame according to the annotated semantic information of the video frame. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is a block diagram of a computer system provided by an exemplary embodiment of the present application;

[0023] Figure 2 is a system framework diagram of a video platform provided by an exemplary embodiment of the present application;

[0024] Figure 3 is a flowchart of a video semantic annotation method provided by an exemplary embodiment of the present application;

[0025] Figure 4 is a schematic diagram of video semantic information annotation provided by an exemplary embodiment of the present application;

[0026] Figure 5 is a flowchart of a video semantic annotation method provided by another exemplary embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a preset plot event provided by an exemplary embodiment of the present application;

[0028] Figure 7 is a schematic diagram of a video playing interface containing semantic annotation information provided by another exemplary embodiment of the present application;

[0029] Figure 8 is a framework diagram of a first virtual role identification method provided by an exemplary embodiment of the present application;

[0030] Figure 9 is a schematic diagram of a virtual element information and plot information extraction method provided by an exemplary embodiment of the present application;

[0031] Figure 10 is a schematic diagram of a video frame type provided by an exemplary embodiment of the present application;

[0032] Figure 11 is a schematic diagram of a video frame containing a game result provided by an example embodiment of the present application;

[0033] Figure 12 is a schematic diagram of a video frame containing game state information provided by an example embodiment of the present application;

[0034] Figure 13 is a flow framework diagram of a video semantic labeling method provided by an example embodiment of the present application;

[0035] Figure 14 is an interface schematic diagram of a live user interface provided by an example embodiment of the present application;

[0036] Figure 15 is a schematic diagram of a virtual environment picture provided by an example embodiment of the present application;

[0037] Figure 16 is a system framework diagram of a video platform provided by another example embodiment of the present application;

[0038] Figure 17 is a block diagram of a video semantic labeling device provided by an example embodiment of the present application;

[0039] Figure 18 is a device structure schematic diagram of a computer device provided by an example embodiment of the present application;

[0040] Figure 19 is a device structure schematic diagram of a server provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0042] First, the terms involved in the embodiments of the present application are introduced:

[0043] Feeds (Web Feed, News Feed, Syndicated Feed): Also named as source material, or feed, or information provider, or supply, or abstract, or source, or news subscription, or web source (English: web feed, news feed, syndicated feed) is a data format. Information publishing platform spreads the latest information to users, usually arranged in a timeline way, which is the most original, most intuitive and most basic display form of feeds. The prerequisite for users to subscribe to a website is that the website provides feeds. Feeds are converged in a place called aggregation, and the software used for aggregation is called aggregator. For end users, the aggregator is a software specially used to subscribe to websites, usually including RSS (Really Simple Syndication) reader, feed reader, news reader, etc.

[0044] Virtual environment: A virtual environment displayed (or provided) by an application when running on a terminal. The virtual environment can be a simulation environment of the real world, a semi-simulation and semi-fictional environment, or a purely fictional environment. The virtual environment can be any one of a two-dimensional virtual environment, a 2.5-dimensional virtual environment, and a three-dimensional virtual environment, which is not limited in the present application. The following embodiments are exemplified by taking the virtual environment as a three-dimensional virtual environment. In some embodiments, the virtual environment is used to provide a battle environment for at least two master virtual characters. The virtual environment includes a symmetrical left lower corner area and a right upper corner area, and the master virtual characters belonging to two opposing camps occupy one of the areas respectively, and the victory target is to destroy the target building, or stronghold, or base, or crystal in the deep part of the opponent's area.

[0045] Virtual character: Refers to a movable object in a virtual environment. The movable object can be a virtual person, a virtual animal, an animation character, etc., such as a person or an animal displayed in a three-dimensional virtual environment. Alternatively, the virtual character is a three-dimensional model created based on animation skeleton technology. Each virtual character has its own shape and volume in a three-dimensional virtual environment, and occupies a part of the space in the three-dimensional virtual environment. The embodiments of the present application take the first virtual character as an example, which refers to one or more first virtual characters in the virtual environment.

[0046] Multiplayer Online Battle Arena (MOBA) games are virtual environments where at least two opposing teams occupy their own map areas and compete against each other, aiming to achieve a specific victory condition. This victory condition includes, but is not limited to: capturing or destroying enemy bases, eliminating enemy virtual characters, surviving within a specified timeframe, acquiring a resource, or achieving a higher score than the opponent within a specified time. MOBA games can be played in rounds, each with a different map. Each virtual team consists of one or more virtual characters, such as 1, 2, 3, or 5. A MOBA game lasts from the start of the game until the victory condition is achieved.

[0047] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0048] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0049] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0050] Cloud Computing: refers to the delivery and use of Internet Technology (IT) infrastructure as a service, and refers to obtaining the required resources in a scalable manner on demand through the network; broad-sense cloud computing refers to the delivery and use of services, and refers to obtaining the required services in a scalable manner on demand through the network. Such services can be IT and software, Internet related, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage, virtualization, load balancing, etc.

[0051] With the development of the Internet, real-time data flow, and diversified connected devices, and the promotion of requirements such as search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly developed. Unlike previous parallel distributed computing, the generation of cloud computing will revolutionize the entire Internet model and enterprise management model from the concept.

[0052] The embodiment of the present application provides a video semantic labeling method, which can store the labeled video in combination with cloud computing technology, and accurately push the labeled video to the corresponding client of the user according to the user's demand.

[0053] The video semantic labeling method based on a virtual environment provided by the embodiment of the present application can be applied to a computer device with strong data processing capability. In a possible implementation manner, the video semantic labeling method based on a virtual environment provided by the embodiment of the present application can be applied to a personal computer, a workstation or a server, that is, the semantic information in the video can be recognized through the personal computer, the workstation or the server, so as to facilitate understanding of the video content. Illustratively, the video semantic labeling method based on a virtual environment is applied to a background server of an application program, so that the terminal installed with the application program can receive the video content containing semantic information by means of the background server.

[0054] Figure 1A schematic diagram of a computer system provided by an example embodiment of the present application is shown. The computer system 100 includes a first terminal 110, a second terminal 111 and a server 120, wherein the first terminal 110 and the second terminal 111 respectively communicate data with the server 120 through a communication network. The communication network can be a wired network or a wireless network, and can be at least one of a local area network, a metropolitan area network and a wide area network.

[0055] An application program is installed and run in the first terminal 110, and the application program is a video playing application program. The application program can be a video application program (including a short video application program), a live broadcast application program, a music application program, a social application program, a virtual reality application program (VR), an augmented reality application program (AR), a game application program, a shopping application program, a payment application program, a group purchase application program, etc. Illustratively, the video application program is installed on the first terminal 110, and the first terminal 110 is a terminal used by a first user, and the first user (a video producer) records a game video and publishes the game video in the video application program.

[0056] An application program is installed and run in the second terminal 111, and the application program is the same application program as the application program in the first terminal 110, or a different application program in the same type. The application program can be a video application program (including a short video application program), a live broadcast application program, a music application program, a social application program, a virtual reality application program (VR), an augmented reality application program (AR), a game application program, a shopping application program, a payment application program, a group purchase application program, etc. Illustratively, the video application program is installed on the second terminal 111, and the second terminal is a terminal used by a second user, and the second user (a video consumer) watches a game video in the video application program, and the game video can be a video published by the first user.

[0057] In some embodiments, the first terminal 110 and the second terminal 111 can be a mobile terminal such as a smart phone, a smart watch, a tablet computer, a laptop computer, a smart robot, etc., or can be a terminal such as a desktop computer, a projection computer, etc., and the type of the terminal is not limited in the embodiments of the present application. It can be understood that the first terminal 110 and the second terminal 111 can be the same terminal, for example, the video producer uses the first terminal 110 to publish a video and uses the first terminal 110 to watch the video, and the video producer is also the video consumer.

[0058] The server 120 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. In a possible implementation, the server 120 is a background server of an application installed in a terminal.

[0059] As shown in Figure 1 In this embodiment, the first terminal 110 sends a video to the server 120, the video is a video containing a MOBA game screen, the server 120 obtains the video 11, and performs multi-modal information extraction 12 from the video 11. Illustratively, the multi-modal information includes virtual element information and game information, the virtual element information includes element information corresponding to active elements and static elements in a virtual environment, and the game information includes information corresponding to the first virtual character in the control. The target plot event matched in at least two plot events based on the combination of multi-modal information in at least two dimensions is determined as the plot event of the video frame 13, and the video frame is labeled according to the plot event of the video frame 13 to obtain semantic labeling information 14 corresponding to the video frame.

[0060] The server 120 sends the labeled video to the second terminal 111, and the video application installed in the second terminal 111 displays a video playing interface of the game video. The video playing interface includes video content of the game video, a game name, a game tag, a highlight clip in the video, and a video of a related special topic corresponding to the game video. The highlight clip and the video of the related special topic are obtained according to the obtained semantic labeling information 14 corresponding to the video. By labeling the semantic information of the game video by the server 120, the user watching the video can quickly capture the exciting moment of the game video, and the user's viewing experience is improved.

[0061] It can be understood that the above embodiment only takes the server corresponding to the video application in the terminal as an example, and in actual application, the above video semantic labeling method can also be applied to the server corresponding to the live application, and the present embodiment does not limit this.

[0062] For convenience of description, each of the following embodiments takes the video semantic labeling method executed by the server corresponding to the application as an example for description.

[0063] Figure 2A system framework diagram of a video platform provided by an example embodiment of the present application is shown. The example is illustrated by taking a game video recorded by a video producer as an example. The video producer uploads the recorded game video to a background server corresponding to a video application (i.e., a video content production end 21). Illustratively, the background server of the video application includes an uplink content interface server 22, a video content storage server 23, a content database server 24, a dispatch center server 25, a statistical interface and analysis server 28, and a downlink content interface server 38.

[0064] The video producer uploads the game video to the uplink content interface server 22, which stores the game video in the video content storage server 23. The uplink content interface server 22 stores meta information of the game video in the content database server 24, which includes at least one of the size of the video file, a cover picture, a video link, a transcoding rate, a file format, a video title, a release time, and an author. The uplink content interface server 22 sends the game video to the dispatch center server 25.

[0065] The dispatch center server 25 calls a video deduplication service 26 to perform deduplication processing on the game video, which includes at least one of title deduplication processing, cover picture deduplication processing, script deduplication processing, video fingerprint deduplication processing, and audio fingerprint deduplication processing. The video deduplication service 26 feeds back a deduplication result to the dispatch center server 25 after performing deduplication processing on the game video.

[0066] The dispatch center server 25 calls an artificial review system 27 to review the deduplicated game video and feeds back a review result to the dispatch center server 25. The artificial review system 27 feeds back the review result to the dispatch center server 25 after reviewing the game video, or the video deduplication service 26 directly sends the deduplicated game video to the artificial review system 27, which reviews the game video and feeds back a review result to the dispatch center server. The artificial review is a preliminary review of whether the video content involves pornography, gambling, or political sensitivity, and then the review personnel need to mark the video content with a label according to the classification or confirm whether the label of the video content is correct.

[0067] The artificial review system 27 sends review information in the process of reviewing the game video to the statistical interface and analysis server 28, which is also used to receive dispatch information of the dispatch center server 25 when performing task dispatching. In some embodiments, the statistical interface and analysis server 28 is used to preliminarily count and analyze the marked sample video and save the counting and analysis result in a video content sample database 33 from which the sample video can be obtained.

[0068] The dispatch center server 25 invokes the video content understanding main service 29 to perform semantic labeling on the video content of the game video. The video content understanding main service 29 includes a video preprocessing service 30, a video information extraction service 31, and a video understanding service 32. The three services are based on a video content processing model 34, and the video content processing model 34 is used to process video frames by invoking the corresponding video content processing model 34 of each service, so as to extract multi-modal information in at least two dimensions from the video frames, and enable the construction of multiple plot events based on the multi-modal information. The video preprocessing service 30 is used to preprocess the game video, such as video frame classification and resolution enhancement of the game video frames; the video information extraction service 31 is used to extract multi-modal information from the game video frames; and the video understanding service 32 is used to obtain plot events generated in a virtual environment according to the extracted multi-modal information, and match the plot events with a target plot event. If the two are matched, the game video is labeled with semantic information according to the target plot event.

[0069] The video content processing model 34 is trained from sample videos obtained from a video content sample database 33.

[0070] The video content understanding main service 29 feeds back the game video labeled with semantic information to the dispatch center server 25, and the dispatch center server 25 sends the video address corresponding to the game video labeled with semantic information to the downstream content interface server 38.

[0071] When a user watches the game video (i.e., a video content consumption end 39), the video address is obtained from the downstream content interface server 38, and the client corresponding to the user obtains the game video from the video content storage server 23 according to the video address. It can be understood that the video addresses in the downstream content interface server 38 are all game videos labeled with semantic information, so that the user can determine the highlight segments in the game video and the related videos of the game video.

[0072] Through the above system framework, the user can quickly obtain the highlight segments in the video, the user can selectively watch the video, and the user can quickly understand the ideas expressed by the video content.

[0073] Figure 3 A flowchart of a video semantic labeling method provided by an example embodiment of the present application is shown. The method is applied to the server 120 shown in FIG. 1. Figure 1 The method includes the following steps:

[0074] Step 301: Obtain a sequence of video frames in a video.

[0075] Illustratively, the server receives a video sent by a terminal, the terminal has an application installed, a user collects a video through the application, or a user collects a video using a camera application in the terminal system, the camera application is invoked by the application to obtain the video, and the recorded video is sent to the server through the application. The application includes a video application (including a short video application) and a live application and the like supporting video collection functions.

[0076] Illustratively, the server has pre-stored videos, or the server obtains videos from a public video dataset.

[0077] The video includes a plurality of video frames forming a video frame sequence. The embodiments of the present application take a game video as an example for illustration, and the video obtained by the server is a video collected from a picture during running of a game application.

[0078] The game application includes the following two pictures during running: a virtual environment picture (a first picture) during game playing and a picture (a second picture) during non-game playing.

[0079] The first picture includes but is not limited to the following pictures: a picture when a first virtual character participates in a game as a player, a picture when the first virtual character watches other virtual characters play as a spectator, a picture of showing a card according to a game strategy, a picture of a user controlling a virtual vehicle (such as a virtual vehicle, a virtual ship, etc.) to drive in a virtual environment, a picture of resisting an attack of a non-user character (Non Player Character, NPC), and a picture of eliminating a preset number of same elements.

[0080] The second picture includes but is not limited to the following pictures: a character selection picture when a user selects a first virtual character to participate in a game, a game loading picture before entering a virtual environment, a skill selection picture of selecting a skill (or a virtual prop) carried by the first virtual character, a message receiving picture of receiving a notification message, a prop purchase picture of purchasing a virtual prop, and a character training picture of simulating the first virtual character to participate in a game.

[0081] The above pictures are pictures contained in video frames in the video, that is, one or more picture frames corresponding to the above pictures in the video frame sequence.

[0082] In step 302, multi-modal information in at least two dimensions is extracted from video frames in the video frame sequence.

[0083] The multi-modal information refers to information with multiple sources or forms. The multi-modal information includes but is not limited to the following types of information:

[0084] When the video frame sequence is a video frame sequence about a MOBA game, the multi-modal information includes information corresponding to virtual characters and game information;

[0085] When the video frame sequence is a video frame sequence about a First Person Shooting Game (FPS), the multi-modal information includes information corresponding to virtual characters and game information;

[0086] When the video frame sequence is a video frame sequence about a racing game, the multi-modal information includes information corresponding to virtual vehicles, time information, and ranking information;

[0087] When the video frame sequence is a video frame sequence about a simulation raising game, the multi-modal information includes information corresponding to raising objects and raising (or management) result information;

[0088] When the video frame sequence is a video frame sequence about a card game, the multi-modal information includes card information and game result information;

[0089] When the video frame sequence is a video frame sequence about a tower defense game, the multi-modal information includes information corresponding to protected objects and information corresponding to NPCs;

[0090] When the video frame sequence is a video frame sequence about a music game, the multi-modal information includes information corresponding to virtual music elements (such as block elements representing notes), score information, and hit information (information generated by continuously hitting multiple virtual music elements).

[0091] Illustratively, a video frame sequence about a MOBA game is taken as an example for description.

[0092] The server extracts information from each video frame in the video frame sequence, such as Figure 4 information 312 is extracted from the video frame 311 in the video frame sequence, the information 312 including virtual element information and game information. The virtual element information refers to information corresponding to virtual elements in a virtual environment, including information corresponding to active elements and static elements displayed in a virtual environment screen. The game information refers to information related to a game of the first virtual character, such as the display of the first virtual character a killing the second virtual character b in the broadcast information control.

[0093] The game information includes game state information and game result information. The game state information is obtained from controls displayed on the user interface. The game state includes skill information or prop information used when the first virtual character attacks another virtual character. For example, a broadcast information control is displayed on the user interface, broadcasting that the first virtual character a kills the second virtual character b. The game result information refers to information corresponding to the game result generated after a game is over.

[0094] Illustratively, the server calls an information extraction model to extract multi-modal information in at least two dimensions from the video frame sequence, the information extraction model being a machine learning model with information extraction capability. The information extraction model can be trained based on an unsupervised learning manner, so that the information extraction model can obtain the trained information extraction model based on less samples.

[0095] Illustratively, the server stores a feature template of the multi-modal information, and matches the multi-modal information in the video frame. If the matching result is consistent, it is determined that the extracted multi-modal information is the multi-modal information corresponding to the video frame.

[0096] Step 303, based on the target plot event matched in the at least two plot events based on the combination of the multi-modal information in at least two dimensions, the plot event of the video frame is determined.

[0097] Among them, the target plot event includes but is not limited to the following events:

[0098] The first type of match event, the first type of match event is used to represent the event of the first virtual role in the virtual environment;

[0099] The second type of match event, the second type of match event is used to represent the event of the virtual vehicle driving in the virtual environment;

[0100] The third type of match event, the third type of match event is used to represent the event of showing cards according to the game strategy;

[0101] The defense damage event is used to represent the event of resisting the damage of the non-user role NPC to the protected object (using virtual props or skills);

[0102] The element elimination event is used to represent the event of eliminating the same elements by dragging the same elements to the same place by a preset number of dragging operations.

[0103] The server combines some plot events according to the multi-modal information extracted from the video frame sequence, compares the plot event with the target plot event, and determines the plot event of the video frame if the two are consistent.

[0104] Illustratively, the video frame sequence is a video frame sequence about a MOBA game. The server extracts the life value information corresponding to the first virtual role and the broadcast information corresponding to the broadcast information control from the video frame sequence. The life value information is that the life value of the first virtual role is lower than the preset life value in a period of time, and the broadcast information is that the first virtual role completes a preset number of kill enemy virtual roles in the period of time. The plot event is named "Silk Blood Multi-Kill".

[0105] Step 304, according to the plot event of the video frame, the video frame sequence is semantically annotated to obtain the semantic information of the video.

[0106] The semantic annotation refers to annotating the category to which each pixel point in the video frame belongs. Through semantic annotation of the video, the information contained in the video content can be extracted, which is convenient for the viewer to understand the video content. The semantic information is used to represent the descriptive information corresponding to the video content.

[0107] As shown in Figure 4 According to the target plot event matched by the combination of the virtual element information and the match information, the video frame sequence is semantically annotated to obtain the video semantic information 313. In some embodiments, the semantic information further includes at least one of the type to which the video content belongs, the user account corresponding to the published video, and the first virtual role, such as user A (account a) publishing a game video about the game type of multi-MOBA, and the plot event generated by the first virtual role in the game is the event corresponding to the game plot.

[0108] In summary, the method provided in the embodiment determines the target plot event matched by the combination of the multi-modal information extracted from the video frame as the plot event of the video frame, thereby using the plot event corresponding to the video frame to semantically annotate the video frame. Without the need for a machine learning model trained by a sample video, the video frame can also be annotated with semantic information, improving the efficiency and accuracy of semantic annotation of the video frame, so that the user can quickly capture the highlight segments in the video frame according to the video frame annotated with semantic information.

[0109] Figure 5 A flowchart of a video semantic annotation method provided by another example embodiment of the application is shown. The method is applied in the server 120 as shown in Figure 1 The method includes the following steps:

[0110] Step 501, obtaining a video frame sequence in a video.

[0111] Illustratively, taking a game video as an example, the game video is a video recorded by a game anchor during live streaming, and the game video is about a MOBA game. The server obtains a video frame sequence from the game video for semantic labeling, and the video frames in the video frame sequence include game screens, which include screens of the game anchor controlling a first virtual character (the first virtual character is an identity participating in a match) to play the match, the game anchor controlling the first virtual character to watch other virtual characters playing the match, the game anchor selecting the first virtual character participating in the match, the game anchor selecting a virtual prop (or skill) carried by the first virtual character, a game loading screen before entering a virtual environment, a screen of purchasing a virtual prop, a screen of simulating the first virtual character participating in the match, and the like.

[0112] The screens of the first virtual characters playing the match further include screens of the first virtual characters attacking each other, screens of reporting or prompting a match status, and screens of the first virtual characters of the same team sending messages to each other. Illustratively, at least one control is displayed on the screens of reporting or prompting the match status, and the control can be a reporting information control or a score control (for counting a match score).

[0113] Illustratively, the game video is about a racing game, and a user controls a virtual vehicle such as a virtual vehicle or a virtual ship to play a speed match, and the match is set with a fixed route, and the virtual vehicle that first rushes through the finish line is the winner. The game video includes at least one of the following screens: a screen of the virtual vehicle driving in a virtual environment, a screen of the virtual vehicle rushing through the finish line, a screen of the user selecting the virtual vehicle, a screen of displaying a ranking corresponding to the match end, a screen of dynamically changing the ranking during the match, and a screen of displaying a speed of the virtual vehicle.

[0114] Illustratively, the game video is about a turret defense game, and a user uses virtual props or virtual elements to resist damage of an NPC to a protected object, or uses virtual props or virtual elements to attack the NPC.

[0115] Step 502, extracting multi-modal information of at least two dimensions from the video frames in the video frame sequence.

[0116] Illustratively, the server extracts the multi-modal information from the video frame sequence by calling an information extraction model, and this embodiment takes an example in which the multi-modal information includes virtual element information and match information. The video frames in the video frame sequence include virtual environment screens and at least one control, and the virtual environment screens include screens of a virtual environment in which a first virtual character is active.

[0117] The first information extraction model is a machine learning model for extracting virtual element information from the virtual environment picture. The first information extraction model can be a classification model, such as a convolutional neural network constructed classification model. The second information extraction model is a machine learning model for extracting game information from the control. The second information extraction model can also be a classification model.

[0118] According to the motion state of the virtual element, the virtual elements are divided into active elements and static elements. The active elements include the first virtual character, the neutral virtual character, and the non-player character (NPC). The static elements include building elements (such as defense towers) in the virtual environment and environment-related elements (such as plant elements and weather elements). The neutral virtual character refers to a virtual character that does not belong to either side of the game. The neutral virtual character corresponds to a health value. The first virtual character of the game can attack the neutral virtual character. Under the attack of the first virtual character, the health value of the neutral virtual character decreases to zero. In some embodiments, when the health value of the neutral virtual character decreases to zero, the neutral virtual character is converted into a reward for the first virtual character to collect. The reward can be currency for purchasing virtual props, skill values for upgrading the first virtual character, virtual props, etc. The NPC refers to a virtual character that belongs to the same team as the first virtual character and is not controlled by the user. The NPC corresponds to a health value and can perform corresponding attack actions according to a preset game logic.

[0119] In one example, the server calls the first information extraction model to extract the role identification of the first virtual character from the virtual environment picture. The server calls the second information extraction model to extract the game information between the first virtual characters from the control, such as the number of other virtual characters killed by the first virtual character a.

[0120] In step 503, the target plot event matched in the at least two plot events based on the combination of the multi-modal information of the at least two dimensions is determined as the plot event of the video frame.

[0121] Illustratively, the target plot event is pre-stored in the server. The event after the combination of the multi-modal information is matched with the target plot event to determine the plot event corresponding to the video frame.

[0122] Step 503 can be replaced by the following steps:

[0123] Step 5031: Obtain the correspondence relationship. The correspondence relationship is used to represent the correspondence between the event after the combination of multimodal information and the target plot event.

[0124] As an illustration, the server stores the correspondence between events composed of multimodal information and target plot events. This correspondence can be at least one of the following: a functional relationship and a lookup table relationship.

[0125] Step 5032: Obtain the target plot event based on the combination of the correspondence and multimodal information, and determine the target plot event as the plot event of the video frame.

[0126] Indicatively, in MOBA game videos, target story events (Type 1 match events) include, but are not limited to, the following events:

[0127] The first match event is used to represent the first virtual character reducing the health of the second virtual character to zero within a preset time period. The health of the first virtual character is lower than the preset health value, and the number of the second virtual characters is a preset number.

[0128] The second match event is used to characterize the duration of the first virtual character's concealed state in the virtual environment exceeding the preset duration, and the first virtual character reducing the second virtual character's health to zero.

[0129] The third match event is used to represent the first virtual character reducing the health of building elements in the virtual environment, and the surrounding range of the building element does not include the second virtual character. The building element is the building element corresponding to the team to which the second virtual character belongs.

[0130] like Figure 6 As shown, virtual element information 314 and match information 315 are extracted from the video frames. Virtual element information includes monster information (neutral virtual character information), turret information (building element information), health information, hero information (information of the first virtual character), minimap information (map preview of the virtual environment), and bush information (scenery information). Monsters are neutral virtual characters in the virtual environment, not belonging to either side of the match; both sides can obtain rewards by killing monsters. Turrets are building elements owned by both sides, with each team possessing at least one turret. Bushes refer to the grass in the virtual environment, used for the first virtual character to hide, allowing them to ambush enemy virtual characters. Match information 315 includes attack status information, match result information, and broadcast information.

[0131] According to the correspondence relationship between the event combined by the virtual element information 314 and the match information 315 and the scenario information flow 316 (preset match event), the scenario information flow 316 that the virtual element information 314 and the match information 315 conform to is determined, and the scenario information flow 316 that conforms is determined as the match event corresponding to the video frame.

[0132] Illustratively, the event combined by the virtual element information 314 and the match information 315 conforms to the first match event 317, the first match event 317 is that the life value (blood bar) of the hero (virtual role) a is lower than the preset life value in a period of time, and the hero a completes a preset number of kills in the period of time, and the first match event 317 is named as silk blood multi-kill.

[0133] Illustratively, the event combined by the virtual element information 314 and the match information 315 conforms to the second match event 318, the second match event 318 is that the hero a stays in the grass for more than a preset time in a period of time, and the hero a completes a kill, and the second match event 318 is named as grass bush squatting person.

[0134] Illustratively, the event combined by the virtual element information 314 and the match information 315 conforms to the third match event 319, the third match event 319 is that the hero a of the A camp hits the crystal (building element) of the B camp alone in a period of time, and there is no hero of the B camp within the range around the crystal of the B camp, and the third match event 319 is named as stealing crystal. The two parties of the match each have a crystal, and after the defense tower on a certain road of the enemy is destroyed, the victory of the game can be obtained by destroying the crystal of the enemy.

[0135] Step 504, determining the start time stamp and the end time stamp of the video clip in the video frame sequence to which the plurality of continuous video frames with the same scenario event belong.

[0136] Information extraction is performed on each video frame in the video frame sequence, when a plurality of continuous video frames represent the same scenario event, the plurality of continuous video frames form a video clip, the time stamp of the video frame corresponding to the start time of the scenario event is obtained, and the time stamp is the start time stamp, and then the time stamp of the video frame corresponding to the end time of the scenario event is obtained, and the time stamp is the end time stamp.

[0137] Step 505, according to the scenario event, the start time stamp and the end time stamp, the video clip is semantically annotated, and the semantic information corresponding to the video clip in the video is obtained.

[0138] With the combination of the start timestamp and the end timestamp, the video segment of the plot event occurring in the video frame sequence can be semantically annotated, such as the semantic information of the game video being the second game event occurring from 2 minutes and 22 seconds to 3 minutes and 3 seconds in the video. In some embodiments, the server also calls the video classification model to identify the video, obtaining the video type, such as game video, food video, science popularization video, funny video, and meme video. The semantic information of the video is obtained by combining the video type and the annotated game event.

[0139] As shown in Figure 7 The annotated video semantic information 400 can be displayed in the video playing interface. The video semantic information 400 includes the game name, the game label, the game theme, and the game event. The game label can be obtained by the information input by the video producer when uploading the video, or by detecting the video frame; the game theme can also be obtained by the information input by the video producer when uploading the video, or by detecting the video frame. Illustratively, the video playing interface also displays the plot event and the time axis of the plot event, and the user can directly jump the playing progress of the playing video to the plot event by clicking the plot event.

[0140] In summary, the method provided in the embodiment determines the matched target plot event as the plot event of the video frame by combining the multi-modal information extracted from the video frame, thereby using the plot event corresponding to the video frame to semantically annotate the video frame. The semantic information of the video frame can be annotated by the machine learning model obtained by training the sample video, improving the efficiency and accuracy of the semantic annotation of the video frame, and enabling the user to quickly capture the highlight segment in the video frame according to the annotated semantic information of the video frame.

[0141] The method provided in the embodiment also one-to-one corresponds the plot event occurring in the video to the time by combining the start timestamp and the end timestamp corresponding to the plot event, facilitating the user to quickly locate the plot event in the video frame according to the timestamp.

[0142] The method provided in the embodiment also refines the multi-modal information and associates the target plot event with the event after the combination of the multi-modal information, extracts the corresponding multi-modal information in the video frame in a corresponding manner for different types of videos, so that the extracted information is more consistent with the video type, thereby matching the target plot event more consistent with the video, and improving the efficiency and accuracy of the semantic annotation of the video.

[0143] Taking the game video as a MOBA game video as an example, how to extract the virtual element information and the game information in the video frame is described.

[0144] I. Extracting virtual element information.

[0145] 1. Virtual element information includes the character identifier corresponding to the first virtual character.

[0146] Step 511: Obtain the first character identifier of the first virtual character participating in the game.

[0147] like Figure 8 As shown, the server obtains the first character identifier 71 of the first virtual character participating in the game through the game loading page. When both sides participate in the game, the server will obtain the first character identifier (i.e., hero name) of the first virtual character selected by each user. For example, user 1 selects hero a, user 2 selects hero b, and user 3 selects hero c. Hero a and hero b are teammates, and hero a and hero c are enemies. Illustratively, the server calls the first information extraction model to extract the first character identifier of each first virtual character from the game loading page; illustratively, after a user selects a first virtual character, the client sends the character identifier of the selected first virtual character to the server, and the server obtains the first character identifier 71 of the first virtual character. In some embodiments, the client sends the game loading page to the server, and the server determines the first character identifier of the first virtual character participating in the game by performing text recognition on the text in the game loading page.

[0148] Step 512: Call the element classification model to identify the health value region corresponding to the first virtual character in the virtual environment screen, and obtain the second character identifier of the first virtual character corresponding to the health value region.

[0149] Indicatively, during the game, the character icon of the first virtual character and the health bar area of ​​the first virtual character are displayed in the same area. Usually, the health bar area is located above the head of the first virtual character, and the character icon of the first virtual character (second character icon 73) is displayed above the health bar area.

[0150] The server uses an element classification model to identify the second character identifier of the first virtual character from game frame images. This element classification model is trained on sample videos pre-labeled with character identifiers. Illustratively, the element classification model first locates the health bar region in the video frame and then identifies the second character identifier of the first virtual character from within that region.

[0151] Step 513: In response to the matching of the first role identifier and the second role identifier, obtain the first confidence level corresponding to the second role identifier.

[0152] The second role identifier 73 is matched with the first role identifier 71 obtained in step 512, and the first confidence 75 between the second role identifier 73 and each first role identifier is calculated. The confidence refers to the degree of matching between the first role identifier and the second role identifier. For example, if the first confidence 75 between the first role identifier 71 and the second role identifier 73 is 0.9, the first role identifier 71 and the second role identifier 73 have a matching probability of 0.9, that is, the first role identifier 71 is close to the second role identifier 73.

[0153] In step 514, a positioning tracking model is called to identify the first virtual role in the virtual environment picture according to the first role identifier, and a third role identifier of the first virtual role displayed in the virtual environment picture is obtained. The third role identifier corresponds to a second confidence, and the positioning tracking model is one-to-one corresponding to the first virtual role in the game.

[0154] The server initializes the positioning tracking model according to the obtained first role identifier 71 for each first virtual role in the video frame. The positioning tracking model will always track the first virtual role. The first virtual role in the video frame is framed, the framed first virtual role is extracted, the third role identifier 72 is extracted according to the framed first virtual role, and the second confidence 74 between the third role identifier 72 and the first role identifier 71 is calculated.

[0155] Illustratively, the positioning tracking model is a tracking model with discriminative prediction ability (Discriminative Model Prediction, Dimp model). The Dimp model has discriminative prediction ability for the tracking target, so as to lock the tracking target in the video frame picture and avoid loss of the tracking target. The positioning accuracy and positioning efficiency of the tracking target are improved, and the subsequent identification of the role identifier of the first virtual role is facilitated.

[0156] The positioning tracking model initializes a tracker for each first virtual role in the initialized virtual environment picture, that is, a tracker is configured for each first virtual role. The tracker tracks the first virtual role according to the role identifier. When the first virtual role leaves the virtual environment picture (for example, the first virtual role exits the current game, or the first virtual role is killed), it is determined whether to track a new target. If the new target is tracked, it is determined whether the new target corresponds to a matching tracker. If the new target has no matching tracker, a new tracker is initialized for the new target. If the new target corresponds to a tracker, the new target and the corresponding tracker are re-matched, and the new target is tracked.

[0157] In step 515, the role identifier of the first virtual role is obtained according to the first confidence and the second confidence.

[0158] The first confidence 75 obtained in step 513 is compared with the second confidence 74 obtained in step 514, and the role identification with higher confidence is selected as the role identification of the first virtual role extracted from the video frame by the server.

[0159] It should be noted that step 513 and step 514 can be executed simultaneously, or step 514 is executed prior to step 513.

[0160] In summary, the method provided in the embodiment extracts the role identification of the first virtual role in the video frame through the dual extraction mode of the positioning tracking model and the classification model, so that the server can accurately extract the role identification of the first virtual role from the video frame, and the result of subsequent semantic annotation of the video frame using the role identification of the first virtual role is more accurate.

[0161] 2. The virtual element information includes position information corresponding to the building element in the virtual environment.

[0162] Step 521, extracting the building element feature corresponding to the building element from the virtual environment picture.

[0163] Illustratively, the server calls the feature extraction model to extract the building element feature from the video frame. Taking the building element as a defensive tower as an example, the extracted building element feature can be the appearance shape of the defensive tower, the position of the defensive tower, and the life value of the defensive tower. The feature extraction model is obtained according to the sample video training, and the sample video includes corresponding building element features.

[0164] Step 522, matching the building element feature with the preset building element feature to obtain a first matching feature.

[0165] Illustratively, the server pre-stores the preset building element feature, or the server includes a feature matching library including the preset building element feature. The building element feature extracted in step 521 is matched with the preset building element feature to determine the first matching feature, which is the feature corresponding to the position information of the building element.

[0166] Step 523, calling the first convolutional neural network to classify the first matching feature to obtain the position information corresponding to the building element.

[0167] The server calls a first convolutional neural network (CNN) to classify the first matching feature, and obtains position information corresponding to the building element. Illustratively, the first convolutional neural network is trained according to a sample video containing preset building element features, the preset building element features including features corresponding to the position information of the building element, and the trained first convolutional neural network can extract the position information corresponding to the building element based on the first matching feature.

[0168] As shown in Figure 9 The server extracts defense tower information 41 from the video frame, the defense tower information 41 including a defense tower position and a defense tower life value. The defense tower position determines the position of the defense tower in the virtual environment, and when the first virtual character is close to the defense tower, the position of the first virtual character can be indirectly obtained. The defense tower life value can determine whether the first virtual character interacts with the defense tower, thereby determining the game event, such as the first virtual character hitting the defense tower.

[0169] In summary, the method provided in the embodiment extracts the position information corresponding to the building element in the virtual environment, determines the position of the building element in the virtual environment according to the position information corresponding to the building element, and when the building element interacts with the first virtual character, more game events can be generated, so that the semantic labeling has more abundant game events (video scripts), and the efficiency and accuracy of semantic labeling of the video frame are improved.

[0170] 3. The virtual element information includes position information corresponding to a neutral virtual character in the virtual environment.

[0171] Step 531: Extracting a character element feature corresponding to a neutral virtual character from a virtual environment picture.

[0172] Similar to the server extracting virtual element information from the video frame, the server also extracts neutral virtual character information from the video frame. The neutral virtual character is an NPC character that does not belong to either side of the game. Illustratively, the server calls a feature extraction model to extract a character element feature corresponding to a neutral virtual character from a video frame, and the feature extraction model is trained according to a sample video containing a character element feature.

[0173] Step 532: Matching the character element feature with a preset character element feature to obtain a second matching feature.

[0174] Illustratively, the server pre-stores preset role element features, or the server includes a feature matching library including the preset role element features. The role element features in step 531 are matched with the preset role element features to determine second matching features, which are features corresponding to the position information of the neutral virtual role.

[0175] In step 533, a second convolutional neural network is called to classify the second matching features to obtain position information corresponding to the neutral virtual role.

[0176] Illustratively, the second convolutional neural network is trained according to a sample video containing preset role element features, the preset role element features including features corresponding to the position information of the neutral virtual role, and the trained second convolutional neural network can extract position information corresponding to the building element based on the second matching features.

[0177] As shown in Figure 9 , the server extracts monster information 42 from the video frame, the monster information 42 including a monster position and a monster life value, the monster position being used to determine a position of the monster in the virtual environment, and when the first virtual role approaches the neutral virtual role, the position of the first virtual role can be indirectly obtained; the monster life value is used to determine whether the first virtual role interacts with the monster, so as to determine a game event, such as the first virtual role killing the monster to obtain a game event corresponding to a reward.

[0178] In summary, the method provided in the embodiment extracts position information corresponding to a neutral virtual role in a virtual environment, indirectly determines a position of a first virtual role in the virtual environment according to the position information corresponding to the neutral virtual role, and when the first virtual role interacts with the neutral virtual role, more game events can be generated, so that semantic labeling has more rich game events (video scripts), and the efficiency and accuracy of semantic labeling of the video frame are improved.

[0179] II. Extracting game information.

[0180] 1. The game information includes game end information.

[0181] In step 541, a video frame classification model is called to classify the video frame sequence to obtain a video frame corresponding to a game end state, the video frame corresponding to the game end state including a control.

[0182] Taking a MOBA game as an example of the game type, the type of the video frame includes at least one of the following types:

[0183] a. A game picture frame, which includes a video frame corresponding to a picture when the first virtual role plays the game.

[0184] like Figure 10 As shown in (a), in the virtual environment, the first virtual character 44 and another virtual character 45 play against each other in the virtual environment. The first virtual character 44 and the other virtual character 45 can be teammates or enemies. The video frames corresponding to the game process are the game screen video frames.

[0185] b. Character selection frame: The character selection frame is used to select the video frame corresponding to the first virtual character participating in the game.

[0186] like Figure 10 As shown in (b), the rectangles 47 on the left represent user accounts belonging to faction A, the rectangles 48 on the right represent user accounts belonging to faction B, and the rectangles 46 at the bottom represent the first virtual character that the user can select. Users can select their first virtual character to participate in the game on this screen; the corresponding video frame is the character selection frame.

[0187] c. End-of-game frame: The end-of-game frame represents the video frame that corresponds to the end of a game.

[0188] like Figure 10 As shown in (c), after a game ends, the game result 49 is displayed on the screen. If the game result 49 is a victory, then the video frame corresponding to that screen is the game end screen frame.

[0189] d. Loading screen frame: The loading screen frame is used to display the video frame corresponding to the first virtual character participating in the game.

[0190] like Figure 10 As shown in (d), before the game begins, i.e. before the first virtual character enters the virtual environment, a game loading screen is displayed. This loading screen shows the participating opposing factions and a character poster 50 for the first virtual character in each faction. In some embodiments, the character poster 50 displays the skin (or clothing) worn by the first virtual character. The video frame corresponding to this game loading screen is a loading screen frame.

[0191] e. Non-target video frames: Non-target video frames are used to represent video frames corresponding to areas of the screen that are not part of the game.

[0192] like Figure 10 As shown in (e), the game video is the video captured during the game streamer's live broadcast. When the game streamer is not playing the game, or when the game streamer enters a non-game interface, a non-target frame is displayed in the live broadcast. For example, if the game streamer pauses the game and displays advertisement 51 in the live broadcast, the video frame corresponding to the live broadcast frame where advertisement 51 is located is a non-target video frame. It can be understood that a non-target video frame can also be a loading frame or a character selection frame.

[0193] The video frame classification model is a machine learning model trained on different types of sample videos. The video frame corresponding to the end of a game refers to the screen displayed after a game ends. Controls are used in this video frame to indicate the end of the current round, such as controls displaying the game result or game leaderboards.

[0194] Step 542: Perform text recognition on the control to obtain the text recognition result.

[0195] like Figure 11 As shown in the left image, the screen 52 displayed after the game ends includes a game-ending control containing text. Recognizing the text in screen 52 yields a text recognition result 53 indicating a victory. This illustrates how a classification model is trained to classify and recognize text in an image, obtaining the text recognition result. The classification model is trained based on sample video frames, including those labeled with text recognition results, such as those labeled "victory," "defeat," or "draw."

[0196] Step 543: In response to the text recognition results including the game results, extract the game end information from the video frame corresponding to the game end state based on the game results.

[0197] In some embodiments, the video frame corresponding to the end of the game also displays "Tap the screen to continue". When "Tap the screen to continue" is recognized, it can also indicate the end of a game. The video frame is used as the end mark of the game, and subsequent video frames will be used as the next game for analysis and understanding.

[0198] In summary, the method of this embodiment extracts game end information from the video frame corresponding to the game end state, and uses this game end information as the end information for the current game, or as the start information for the next game. This serves to separate two adjacent games, enabling the server to accurately determine the start and end times of a game.

[0199] 2. Match information includes match status information between the first virtual characters.

[0200] Step 551: Call the video classification model to classify the video frame sequence to obtain the game screen frames, which include controls.

[0201] Similarly, the server uses a video frame classification model to identify frames in the video sequence, determining the game frames from the sequence. The video frame classification model is a machine learning model trained on different types of sample videos. Game frames include the video frames corresponding to the scenes when the first virtual character is playing a match.

[0202] Illustratively, the pair of game picture frames contains information broadcast controls for the game state, such as broadcasting that the first virtual character killed two virtual characters, or the game score, or the gain score.

[0203] Step 552, identify the hit state in the control to obtain a hit state result, the hit state result being the corresponding event result when the hit event occurs between the first virtual characters.

[0204] Illustratively, the server calls the classification model to identify the region in the game picture frame where the hit state occurs, i.e., to identify the first virtual character. The classification model is trained according to a sample video containing a hit event. As shown in Figure 12 The first virtual character 54 hits the neutral virtual character 55.

[0205] When a hit event occurs between the first virtual characters, a broadcast information control is usually displayed in the user interface, and the game state information is obtained by identifying the broadcast information control.

[0206] A hit refers to a first virtual character hitting another virtual character, or a first virtual character hitting a neutral virtual character, and the hit can reduce the life value of the attacked object. The classification model is obtained by training a sample video containing a hit state annotation, and is used to frame the virtual character in the video frame that produces a hit event.

[0207] Step 553, extract the game state information from the game picture frame according to the hit state result.

[0208] As shown in Figure 9 The game state information can be obtained through the broadcast information control 43, such as the classification model identifying that the first virtual character a in the A camp kills the first virtual character b in the B camp, and the server obtains the game state information as the first virtual character a killing the first virtual character b.

[0209] In summary, the method provided in this embodiment identifies the event result of the hit event occurring between the first virtual characters to obtain a hit state result, thereby extracting the game information between the first virtual characters from the game picture frame according to the hit state result, and through the game information between the first virtual characters, the server can combine more comprehensive information to perform semantic annotation on the video frame, thereby improving the efficiency and accuracy of semantic annotation on the video frame.

[0210] Figure 13 A flow framework diagram of a video semantic annotation method provided by an example embodiment of the present application is shown. The method can be applied to, for example Figure 1The server 120 shown. The flow includes the following three parts: video preprocessing 56, video information extraction 57 and video semantic labeling 58.

[0211] I. Video preprocessing.

[0212] Video preprocessing 56 refers to the video published by the user (video producer) is frame cutting processing 561, game frame classification processing 562 and resolution enhancement processing 563.

[0213] 1. Frame cutting processing 561:

[0214] Illustratively, the game video collected in the process of game anchor live broadcast is taken as an example, as shown in Figure 14 Some game anchors record live video in the form of picture-in-picture during live broadcast, that is, add a frame to the virtual environment picture (game interface). In this case, if the subsequent identification task is directly performed, the identification result will be greatly disturbed, so it is necessary to identify the game video area first to improve the labeling accuracy and identification efficiency when labeling the semantics of the game video.

[0215] The video frames in the video frame sequence include a virtual environment picture area and a live picture area, the live picture area is used to represent the picture area for shooting the live broadcast process of the anchor, and the live picture area does not include the virtual environment picture area. First, determine the boundary between the virtual environment picture area and the live picture area; then, according to the boundary, the live picture area is cropped to obtain the video frame containing the virtual environment picture area.

[0216] The boundary between the virtual environment picture and the live picture area is determined in the following way:

[0217] S1, obtain the video frame sequence after binaryzation processing.

[0218] Binaryzation processing refers to setting the gray value of the pixel point in the video frame to 0 or 255, that is, converting the entire video frame to black and white. The game video is binaryzation processed to obtain the binaryzation processed video frame sequence.

[0219] S2, according to the Huffman algorithm, the pixel points contained in the video frames in the binaryzation processed video frame sequence are converted to Huffman space.

[0220] Each pixel point contained in the binaryzation processed video frame is converted to Huffman space according to the Huffman algorithm, wherein each pixel point in the Huffman space corresponds to a curve, and when the curves intersect together, it means that the pixel points corresponding to the multiple curves are on the same curve in the video frame. Therefore, according to the intersecting curves in the Huffman space, the curve formed by each pixel point in the binaryzation processed video frame can be determined.

[0221] S3, in response to the number of curves intersecting at the same pixel point in the Hough space being greater than or equal to a number threshold, determining that there is a straight line corresponding to the pixel point in the video frame.

[0222] In theory, a pixel point corresponds to an infinite number of curves or curves in any direction. In actual application, the number of curves (i.e. a limited number of directions) is limited for calculation, so when the number of curves passing through a certain intersection point in the Hough space exceeds the number threshold, the intersection point corresponds to a curve (or a straight line) in the video frame.

[0223] S4, determining the boundary between the virtual environment picture area and the live picture area according to the straight line corresponding to the pixel point.

[0224] As shown in Figure 14 The boundary 61 is a straight line determined by the Hough algorithm, which divides the virtual environment picture 62 and the first live picture area 63 and the second live picture area 64 according to the straight line, so as to determine the virtual environment picture area in the video frame. The first live picture area 63 is the picture area corresponding to the host's avatar, and the second live picture area 64 is the picture area for real-time shooting of the host.

[0225] 2, game frame classification processing 562:

[0226] Because the video sources published by the video producer come from various channels, the quality is uneven, and in the game video, a large number of non-target frames are usually included. As shown in Figure 10 During the game running, the game video includes virtual environment picture frames, host chat picture frames, game non-game picture frames (such as store interface, hall interface, transaction interface corresponding video frames, etc.), different picture frames contain different information, such as character selection picture frame and loading picture frame contain the character identification of the first virtual character participating in the game, game end picture frame contains the game result information, game picture frame contains the hitting state information in the game process, etc. By filtering out non-target video frames in advance, using corresponding identification mode for specific category game frame, the processing efficiency of the video will be improved and the calculation resources will be saved.

[0227] Illustratively, a classification model constructed by a convolutional neural network trained based on sample images in the ImageNet database is used to train the classification model by labeling the video type, so as to realize the identification of the target video frame in the game video. ImageNet database is a large visual database for visual object recognition software research. More than 14 million uniform resource locators (URLs) of images are manually annotated by ImageNet database to indicate the objects in the pictures, and some images also include boundary boxes.

[0228] 3. resolution enhancement processing 363:

[0229] First, a backbone network Backbone is called to extract features of the video frames in the sequence of video frames, to obtain video features corresponding to the video frames; then, an enhanced super-resolution generative adversarial network ESRGAN is called to process the video features, to obtain the video frames with enhanced resolution.

[0230] Due to the encoding and decoding operations in the video transmission process, the game video resolution published by the video producer is not very high, generally around 720p, and for extracting small map and other small size control information, a large error will be generated. The embodiment of the application extracts video features from the video frames through the Backbone network, and processes the video features through the enhanced super-resolution generative adversarial network (ESRGAN). Since the ESRGAN network can enhance the resolution of the video, the video frames with enhanced resolution are obtained, and in the subsequent process of extracting multi-modal information, the extraction can be based on the video frames with high definition, so that the information extraction is more accurate. The ESRGAN network and the Backbone network are relatively mature methods for enhancing the resolution of pictures in the art, and will not be described here.

[0231] Super-resolution imaging (SR or SRI) is a technique for improving image resolution. Super-resolution imaging techniques are used in image processing and super-high resolution microscopes.

[0232] Generative adversarial network (English: Generative Adversarial Network, abbreviated as GAN) is a method of unsupervised learning, which learns through the mutual game of two neural networks. The generative adversarial network is composed of a generative network and a discriminative network. The generative network randomly samples from the latent space as input, and the output result needs to imitate the real samples in the training set as much as possible. The input of the discriminative network is the real sample or the output of the generative network, and its purpose is to distinguish the output of the generative network from the real sample as much as possible. The generative network wants to "deceive" the discriminative network as much as possible. The two networks are in constant adjustment of parameters, and the ultimate goal is to make the discriminative network unable to judge whether the output result of the generative network is real.

[0233] Figure 15 (a) is a small map picture in the video uploaded by the video publisher, Figure 15the effect of the resolution enhancement of the small map by (b) provided in the embodiments of the present application, Figure 15 the resolution of the small map of (b) is better than Figure 15 the resolution of the small map of (a).

[0234] II. Video information extraction.

[0235] Video information extraction 57 refers to extracting virtual element information and game information from a video. The embodiments of the present application extract virtual element information and game information in a video through a video information extraction service 31 as shown in Figure 2 The information extraction method has been described in detail in the above embodiments, and will not be described here.

[0236] The extracted information is written into a distributed file storage database (mongodb database). The mongodb database supports multi-language queries, geographic location queries, and has the characteristics of flexible data structure, so as to realize efficient use of the extracted information. Taking a game video as an example, the information extracted by the video information extraction service 31 includes health information (including the health of the first virtual character, the neutral virtual character, and the defense tower), broadcast information corresponding to the broadcast control, defense tower information, monster information, grass information, small map information, character identification of the first virtual character, game result information, game state information, etc.

[0237] Different information extraction services correspond to different information extraction services. The different information extraction services are standardized according to the video content processing model. Standardization means that the video content processing model is constructed in the service framework, that is, the weight parameters corresponding to the video content processing model are added to the service framework. When using the information extraction service, the actual input parameters (such as video or extracted element features from the video) need to be provided to the server, so as to obtain the virtual element information and game information of the video.

[0238] As shown in Figure 16 The video content processing model 34 is obtained through the video frame extraction service 35 and the sample video in the video content sample library 33. The background server of the video application program further includes a download file system 36. The download file system 36 is used to download and obtain the original video content from the video content storage server 23, control the speed and progress of the download, and is usually a group of parallel servers composed of related task scheduling and distribution clusters; the video file downloaded is called extraction frame extraction service 35 to obtain the necessary video information from the video source file, that is, to extract virtual element information and game information, as information for subsequent construction of the video content processing model 34.

[0239] III. Video semantic annotation.

[0240] Video semantic annotation 58 refers to semantic annotation of the video to obtain semantic annotation information, which is used to describe the video content.

[0241] The video semantic annotation 58 annotates the video by calling the video understanding service 32 as shown in Figure 2 The video understanding service fuses the video multi-modal information (including virtual element information and game information) extracted in the video information extraction service 31, analyzes the plot appearing in the game video through the pre-edited script logic (preset game event), recalls the corresponding game event when the preset script condition is met, and thus marks the highlight clip.

[0242] Illustratively, when directly using the information extracted in the video information extraction service 31, there are special effects, occlusions and other problems in the game video, which cannot avoid the problems of missed detection and false detection, which leads to very low generalization and low precision of the script logic. Therefore, the information extracted in the video information extraction service 31 is first post-processed. The post-processing includes smoothing and closure processing, and the information extracted from the adjacent two frames is verified with each other, which can further reduce the influence of false detection and missed detection caused by occlusion and special effects and other problems.

[0243] The smoothing processing refers to inserting the lost frames into the decoded video to form a smooth video and improve the quality of the video.

[0244] Closure refers to a function that can read the internal variables of other functions, that is, the variables defined in the function content can be accessed within the internal scope of the function itself, and cannot be accessed outside the function. Closure is a bridge for the function outside to read the internal variables of the function, so that private local variables can be shared by multiple functions. The closure processing extracts video information by combining multiple video frames.

[0245] The information extracted from the video is stored in the mongdb database, and these information can be used as basic elements to obtain a large number of complex video scripts (preset game events) by continuously editing new scripts and configuring processing flow.

[0246] In summary, a large amount of semantic annotation is completed by extracting virtual element information and game information from the video without manual annotation; the efficiency and accuracy of information extraction are improved by video frame cutting, game frame classification, and resolution enhancement; the error is reduced by using computer vision-based method for video basic information extraction and smoothing and closure processing; most of the algorithm techniques use unsupervised learning method without annotation data. For scenes that are difficult to solve by some unsupervised learning methods, synthetic sample data is used so that semantic annotation of the video can be achieved without machine learning model.

[0247] The following is an embodiment of the device of the present application. For details not described in detail in the device embodiment, reference can be made to the corresponding description in the above method embodiment. Herein, no further description is given.

[0248] Figure 17 A structural diagram of a video semantic labeling device provided by an example embodiment of the present application is shown. The device can be realized by software, hardware or a combination of both to become all or part of a terminal. The device includes:

[0249] The acquisition module 1710 is configured to acquire a video frame sequence in a video.

[0250] The extraction module 1720 is configured to extract multi-modal information in at least two dimensions from a video frame in the video frame sequence.

[0251] The processing module 1730 is configured to determine a target plot event in at least two plot events based on the multi-modal information in at least two dimensions matched by the combination of the multi-modal information, as a plot event of the video frame.

[0252] The labeling module 1740 is configured to perform semantic labeling on the video frame sequence according to the plot event of the video frame, to obtain semantic information of the video.

[0253] In an optional embodiment, the labeling module 1740 is configured to determine a start timestamp and an end timestamp of a video segment to which a plurality of continuous video frames with the same plot event belong in the video frame sequence; and perform semantic labeling on the video segment according to the plot event, the start timestamp and the end timestamp, to obtain semantic information corresponding to the video segment in the video.

[0254] In an optional embodiment, the acquisition module 1710 is configured to acquire a corresponding relationship, the corresponding relationship being used to represent a corresponding relationship between an event after combination of the multi-modal information and the target plot event; and acquire the target plot event according to the corresponding relationship and the event after combination of the multi-modal information, and determine the target plot event as the plot event of the video frame.

[0255] In an optional embodiment, a video frame in the video frame sequence includes a virtual environment picture and at least one control, and the virtual environment picture includes a picture of a virtual environment in which a first virtual character is active.

[0256] The extraction module 1720 is configured to call a first information extraction model to extract virtual element information from the virtual environment picture, the virtual element information including information corresponding to at least one of active elements and static elements in the virtual environment picture; and call a second information extraction model to extract session information from the control, the session information including information corresponding to the first virtual character in the control.

[0257] In an optional embodiment, the virtual element information includes a role identifier corresponding to the first virtual role, and the first information extraction model includes an element classification model and a positioning tracking model.

[0258] The acquisition module 1710 is configured to acquire a first role identifier of a first virtual role participating in the game;

[0259] The extraction module 1720 is configured to call the element classification model to identify a health value region corresponding to the first virtual role in the virtual environment picture, to obtain a second role identifier of the first virtual role corresponding to the health value region.

[0260] The processing module 1730 is configured to, in response to the first role identifier matching the second role identifier, obtain a first confidence degree corresponding to the second role identifier.

[0261] The extraction module 1720 is configured to call the positioning tracking model to identify the first virtual role in the virtual environment picture according to the first role identifier, to obtain a third role identifier of the first virtual role displayed in the virtual environment picture, the third role identifier corresponding to a second confidence degree, the positioning tracking model corresponding to the first virtual role participating in the game in a one-to-one manner; and obtain the role identifier of the first virtual role according to the first confidence degree and the second confidence degree.

[0262] In an optional embodiment, the virtual element information includes position information of a building element in the virtual environment, and the first information extraction model includes a first convolutional neural network.

[0263] The extraction module 1720 is configured to extract building element features corresponding to the building element from the virtual environment picture; the processing module 1730 is configured to match the building element features with preset building element features to obtain first matching features; and the extraction module 1720 is configured to call the first convolutional neural network to classify the first matching features to obtain the position information of the building element.

[0264] In an optional embodiment, the virtual element information includes position information of a neutral virtual role in the virtual environment, and the first information extraction model includes a second convolutional neural network.

[0265] The extraction module 1720 is configured to extract role element features corresponding to the neutral virtual role from the virtual environment picture; the processing module 1730 is configured to match the role element features with preset role element features to obtain second matching features; and the extraction module 1720 is configured to call the second convolutional neural network to classify the second matching features to obtain the position information of the neutral virtual role.

[0266] In an optional embodiment, the game information includes game end information, and the second information extraction model includes a video frame classification model.

[0267] The extraction module 1720 is configured to invoke the video frame classification model to classify the video frame sequence to obtain a video frame corresponding to a game end state, the video frame corresponding to the game end state includes a control; perform text recognition on the control to obtain a text recognition result; and in response to the text recognition result including game result, obtain the game end information according to the game result.

[0268] In an optional embodiment, the game information includes game state information between the first virtual characters, and the second information extraction model includes a video frame classification model.

[0269] The extraction module 1720 is configured to invoke the video frame classification model to classify the video frame sequence to obtain a game picture frame, the game picture frame includes a control; identify a hitting state in the control to obtain a hitting state result, the hitting state result being an event result corresponding to a hitting event between the first virtual characters; and obtain the game state information according to the hitting state result.

[0270] In an optional embodiment, the video frames in the video frame sequence include a virtual environment picture region and a live picture region, the live picture region being used to represent a picture region for shooting a live process of the host, and the live picture region does not include the virtual environment picture region.

[0271] The processing module 1730 is configured to determine a boundary between the virtual environment picture region and the live picture region; and crop the live picture region according to the boundary to obtain a video frame containing the virtual environment picture region.

[0272] In an optional embodiment, the acquisition module 1710 is configured to acquire a video frame sequence after binaryzation processing; and the processing module 1730 is configured to convert pixel points contained in a video frame in the video frame sequence after binaryzation processing to a Huffman space according to a Huffman algorithm; in response to a number of curves intersecting at a same pixel point in the Huffman space being greater than or equal to a number threshold, determine that a straight line corresponding to the pixel point exists in the video frame; and determine a boundary between the virtual environment picture region and the live picture region according to the straight line corresponding to the pixel point.

[0273] In an optional embodiment, the extraction module 1720 is configured to invoke a backbone network Backbone to perform feature extraction on a video frame in the video frame sequence to obtain a video feature corresponding to the video frame; and the processing module 1730 is configured to invoke an enhanced super-resolution generative adversarial network ESRGAN to process the video feature to obtain a resolution-enhanced video frame.

[0274] To sum up, the device provided in this embodiment determines the matched target plot event combined by the multi-modal information extracted from the video frame as the plot event of the video frame, so as to perform semantic labeling on the video frame by using the plot event corresponding to the video frame, and the semantic information of the video frame can also be labeled by using the machine learning model trained by the sample video, thereby improving the efficiency and accuracy of semantic labeling on the video frame, and enabling the user to quickly capture the wonderful segments in the video frame according to the video frame with labeled semantic information.

[0275] The device provided in this embodiment also one-to-one corresponds the plot event occurring in the video to the time by combining the start timestamp and the end timestamp corresponding to the plot event, so as to facilitate the user to quickly locate the plot event in the video frame according to the timestamp.

[0276] The device provided in this embodiment also refines the multi-modal information and associates the multi-modal information with the event after the multi-modal information is combined with the target plot event, extracts the corresponding multi-modal information in the video frame in a corresponding manner for different types of videos, so that the extracted information is more consistent with the video type, thereby matching the target plot event more consistent with the video, and improving the efficiency and accuracy of semantic labeling on the video.

[0277] The device provided in this embodiment also identifies the role identifier of the first virtual role in the video frame by using the dual extraction mode of the positioning tracking model and the classification model, so that the server can accurately extract the role identifier of the first virtual role from the video frame, and the result of subsequent semantic labeling on the video frame by using the role identifier of the first virtual role is more accurate.

[0278] The device provided in this embodiment also extracts the position information of the building element in the virtual environment, determines the position of the building element in the virtual environment according to the position information of the building element, and when the building element interacts with the first virtual role, more game events can be generated, so that the semantic labeling has more rich game events (video scripts), and the efficiency and accuracy of semantic labeling on the video frame are improved.

[0279] The device provided in this embodiment also extracts the position information of the neutral virtual role in the virtual environment, indirectly determines the position of the first virtual role in the virtual environment according to the position information of the neutral virtual role, and when the first virtual role interacts with the neutral virtual role, more game events can be generated, so that the semantic labeling has more rich game events (video scripts), and the efficiency and accuracy of semantic labeling on the video frame are improved.

[0280] The device provided in the embodiment also extracts game end information from the video frame corresponding to the game end state, takes the game end information as the end information corresponding to the current game, and also takes the game end information as the start information corresponding to the next game, plays a segmentation role between the two adjacent games, and enables the server to accurately determine the start time and the end time of a game.

[0281] The device provided in the embodiment also identifies the event result of the hitting event between the first virtual characters, obtains a hitting state result, extracts the game information between the first virtual characters from the game picture frame according to the hitting state result, and enables the server to perform semantic labeling on the video frame in combination with more comprehensive information, thereby improving the efficiency and the accuracy of semantic labeling on the video frame.

[0282] The device provided in the embodiment also determines the boundary between the virtual environment picture area and the live picture area in the video frame by using the Huffman algorithm, enables the server to crop the live picture area in the video frame, retains the virtual environment picture area, focuses on identifying the content in the virtual environment picture, and improves the information extraction speed of the server on the video frame and the labeling efficiency of the semantic information.

[0283] The device provided in the embodiment also improves the resolution of the video by using the ESRGAN network and the Backbone network, extracts information based on the video with higher definition, and makes the extracted multi-modal information more accurate.

[0284] It should be noted that the video semantic labeling device provided in the above embodiment is only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the video semantic labeling device and the video semantic labeling method provided in the above embodiment belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0285] Figure 18A structural block diagram of a computer device 1800 provided by an example embodiment of the present application is shown. The computer device 1800 can be a terminal in which a live client is installed in the above-described embodiments, and can be a smart phone, a tablet computer, an MP3 player, an MP4 player, and the like. The computer device 1800 can also be referred to as a user device, a portable terminal, a smart watch, a smart robot, a smart speaker, and the like.

[0286] Generally, the computer device 1800 includes a processor 1801 and a memory 1802.

[0287] The processor 1801 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1801 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1801 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1801 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content to be displayed on a display screen. In some embodiments, the processor 1801 can further include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0288] The memory 1802 can include one or more computer-readable storage media. The computer-readable storage media can be tangible and non-transitory; a computer-readable storage medium does not include a transitory signal. In some embodiments, the memory 1802 includes high-speed random access memory and can include nonvolatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other nonvolatile solid-state storage devices. In some embodiments, the memory 1802 stores instructions for execution by the processor 1801. In some embodiments, the memory 1802 stores data used in accordance with the instructions stored by the memory 1802.

[0289] In some embodiments, the computer device 1800 can optionally include a peripheral interface 1803 and one or more peripheral devices. Specifically, the peripheral devices include at least one of a radio frequency circuit 1804, a touch display screen 1805, a camera component 1806, an audio circuit 1807, a positioning component 1808, and a power supply 1809.

[0290] The peripheral interface 1803 can be used to connect the I / O (input / output) related peripheral device to the processor 1801 and the memory 1802. In some embodiments, the processor 1801, the memory 1802, and the peripheral interface 1803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1801, the memory 1802, and the peripheral interface 1803 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this respect.

[0291] The radio frequency circuit 1804 is used to receive and transmit RF (radio frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1804 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1804 can also include NFC (Near Field Communication) related circuitry, and the present application is not limited in this respect.

[0292] The touch display screen 1805 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. The touch display screen 1805 is further configured to capture touch signals on or above the surface of the touch display screen 1805. The touch signals can be input to the processor 1801 as control signals for processing. The touch display screen 1805 is configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the touch display screen 1805 can be one, configured to set the front panel of the computer device 1800; in other embodiments, the touch display screen 1805 can be at least two, respectively configured to set different surfaces of the computer device 1800 or in a folding design; in other embodiments, the touch display screen 1805 can be a flexible display screen, configured to set a curved surface or a folding surface of the computer device 1800. Even, the touch display screen 1805 can also be configured to be an irregular shape other than a rectangle, i.e., a notched screen. The touch display screen 1805 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0293] The camera assembly 1806 is configured to capture images or videos. Optionally, the camera assembly 1806 includes a front camera and a rear camera. Generally, the front camera is configured to implement video calls or selfies, and the rear camera is configured to implement photo or video shooting. In some embodiments, the rear camera is at least two, respectively configured to be any one of a main camera, a depth-of-field camera, and a wide-angle camera, to implement the background blur function by fusing the main camera and the depth-of-field camera, and to implement the panoramic shooting and VR (Virtual Reality) shooting functions by fusing the main camera and the wide-angle camera. In some embodiments, the camera assembly 1806 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0294] The audio circuit 1807 is configured to provide an audio interface between a user and the computer device 1800. The audio circuit 1807 can include a microphone and a speaker. The microphone is configured to collect sound waves from the user and the environment, and convert the sound waves into an electrical signal input to the processor 1801 for processing, or input to the radio frequency circuit 1804 for voice communication. The microphone can be multiple microphones arranged at different positions of the computer device 1800 for stereo sound collection or noise reduction. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is configured to convert an electrical signal from the processor 1801 or the radio frequency circuit 1804 into sound waves. The speaker can be a traditional thin-film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals into sound waves audible to humans, and also convert electrical signals into sound waves inaudible to humans for ranging purposes. In some embodiments, the audio circuit 1807 can further include a headphone jack.

[0295] The positioning component 1808 is configured to locate the current geographic position of the computer device 1800 for navigation or LBS (Location Based Service). The positioning component 1808 can be a positioning component based on the GPS (Global Positioning System) of the United States, the Beidou system of China, or the Galileo system of Russia.

[0296] The power supply 1809 is configured to supply power to various components in the computer device 1800. The power supply 1809 can be AC power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1809 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery that is charged through a wired line, and the wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0297] In some embodiments, the computer device 1800 further includes one or more sensors 1810. The one or more sensors 1810 include, but are not limited to, an acceleration sensor 1811, a gyroscope sensor 1812, a pressure sensor 1813, a fingerprint sensor 1814, an optical sensor 1815, and a proximity sensor 1816.

[0298] The acceleration sensor 1811 is configured to detect the acceleration in three coordinate axes of a coordinate system established by the computer device 1800. For example, the acceleration sensor 1811 is configured to detect the components of the gravitational acceleration in the three coordinate axes. The processor 1801 can control the touch display screen 1805 to display a user interface in a landscape view or a portrait view according to the gravitational acceleration signals collected by the acceleration sensor 1811. The acceleration sensor 1811 can be used for games or collection of motion data of a user.

[0299] The gyroscope sensor 1812 is configured to detect the orientation and rotation angle of the computer device 1800. The gyroscope sensor 1812 can be used together with the acceleration sensor 1811 to collect 3D motions of a user with respect to the computer device 1800. The processor 1801 can implement the following functions according to the data collected by the gyroscope sensor 1812: motion sensing (e.g., changing a UI according to a tilt operation of a user), image stabilization during shooting, game control, and inertial navigation.

[0300] The pressure sensor 1813 can be disposed on the side frame of the computer device 1800 and / or under the touch display screen 1805. When the pressure sensor 1813 is disposed on the side frame of the computer device 1800, the pressure sensor 1813 can detect a holding signal of a user with respect to the computer device 1800, and perform left-hand or right-hand recognition or a shortcut operation according to the holding signal. When the pressure sensor 1813 is disposed under the touch display screen 1805, the pressure sensor 1813 can control an operable control on a UI according to a pressure operation of a user with respect to the touch display screen 1805. The operable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0301] The fingerprint sensor 1814 is configured to collect a fingerprint of a user, and identify an identity of the user according to the collected fingerprint. When the identity of the user is identified as a trusted identity, the processor 1801 authorizes the user to perform a related sensitive operation, which includes unlocking a screen, viewing encrypted information, downloading software, payment, and changing a setting, etc. The fingerprint sensor 1814 can be disposed on the front, back, or side of the computer device 1800. When a physical button or a vendor logo is disposed on the computer device 1800, the fingerprint sensor 1814 can be integrated with the physical button or the vendor logo.

[0302] The optical sensor 1815 is configured to collect ambient light intensity. In one embodiment, the processor 1801 can control the display brightness of the touch display screen 1805 according to the ambient light intensity collected by the optical sensor 1815. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1805 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1805 is decreased. In another embodiment, the processor 1801 can also dynamically adjust the shooting parameters of the camera assembly 1806 according to the ambient light intensity collected by the optical sensor 1815.

[0303] The proximity sensor 1816, also referred to as a distance sensor, is usually arranged on the front face of the computer device 1800. The proximity sensor 1816 is configured to collect the distance between the user and the front face of the computer device 1800. In one embodiment, when the proximity sensor 1816 detects that the distance between the user and the front face of the computer device 1800 gradually decreases, the processor 1801 controls the touch display screen 1805 to switch from the bright screen state to the off-screen state; when the proximity sensor 1816 detects that the distance between the user and the front face of the computer device 1800 gradually increases, the processor 1801 controls the touch display screen 1805 to switch from the off-screen state to the bright screen state.

[0304] Those skilled in the art can understand that the structure shown in FIG. 10 is not a limitation on the computer device 1800, and the computer device 1800 can include more or fewer components than those shown in the figure, or combine certain components, or adopt a different arrangement of components. Figure 19 Those skilled in the art can understand that the structure shown in FIG. 10 is not a limitation on the computer device 1800, and the computer device 1800 can include more or fewer components than those shown in the figure, or combine certain components, or adopt a different arrangement of components.

[0305] Figure 1 FIG. 11 shows a structure schematic diagram of a server provided by an example embodiment of the present application. The server can be, for example, ​ the server 120 in the computer system 100 shown in FIG. 10.

[0306] The server 1900 includes a central processing unit (CPU) 1901, a system memory 1904 including a random access memory (RAM) 1902 and a read-only memory (ROM) 1903, and a system bus 1905 connecting the system memory 1904 and the central processing unit 1901. The server 1900 also includes a basic input / output system (I / O system) 1906 that helps transfer information between various devices in the computer, and a mass storage device 1907 for storing an operating system 1913, application programs 1914, and other program modules 1915.

[0307] The basic input / output system 1906 includes a display 1908 for displaying information and input devices 1909 such as a mouse, keyboard, or the like for inputting information by a user. The display 1908 and the input devices 1909 are connected to the central processing unit 1901 through an input / output controller 1910 connected to the system bus 1905. The basic input / output system 1906 can also include the input / output controller 1910 for receiving and processing input from a number of other devices, including a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1910 provides output to a display screen, a printer, or other type of output device.

[0308] The mass storage device 1907 is connected to the central processing unit 1901 through a mass storage controller (not shown) connected to the system bus 1905. The mass storage device 1907 and its associated computer-readable media provide non-volatile storage for the server 1900. That is, the mass storage device 1907 can include a computer-readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0309] The computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, erasable programmable read only memory (EPROM, EEPROM), flash memory or other solid state memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Random access memory can include resistance random access memory (ReRAM) and dynamic random access memory (DRAM). Of course, the computer-readable media is not limited to the foregoing examples. The system memory 1904 and the mass storage device 1907 described above can be collectively referred to as memory.

[0310] According to various embodiments of the present application, the server 1900 can also be operated to run on a remote computer connected to the network through a network connection such as the Internet. That is, the server 1900 can be connected to the network 1912 through the network interface unit 1911 connected to the system bus 1905, or can be connected to other types of network or remote computer systems (not shown) using the network interface unit 1911.

[0311] The above-described memory also includes one or more programs, one or more programs stored in the memory configured to be executed by the CPU.

[0312] In an optional embodiment, a computer device is provided, which includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the video semantic labeling method as described above.

[0313] In an optional embodiment, a computer readable storage medium is provided, which stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the video semantic labeling method as described above.

[0314] Optionally, the computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a solid state disk (SSD), an optical disk, etc. Among them, the random access memory can include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The above-mentioned application embodiment serial number is only for description, not representing the pros and cons of the embodiments.

[0315] The application embodiments also provide a computer device, which includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the video semantic labeling method provided by the above-mentioned method embodiments.

[0316] The embodiment of the present application further provides a computer readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the video semantic labeling method provided by each method embodiment.

[0317] The embodiment of the present application further provides a computer program product or a computer program, wherein the computer program product or the computer program comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video semantic labeling method as described in the above aspects.

[0318] It should be understood that "multiple" mentioned herein refers to two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0319] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0320] The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for video semantic annotation, characterized in that, The method comprises: obtaining a video frame sequence in a video, a video frame in the video frame sequence comprising a virtual environment picture and at least one control, the virtual environment picture comprising a picture of a virtual environment in which a first virtual character is active; extracting multi-modal information in at least two dimensions from the video frames in the video frame sequence, the extracting multi-modal information in at least two dimensions from the video frames in the video frame sequence comprising: calling a first information extraction model to extract virtual element information from the virtual environment picture, the virtual element information comprising information corresponding to at least one of active elements and static elements in the virtual environment picture; calling a second information extraction model to extract session information from the control, the session information comprising information corresponding to the first virtual character in the control; determining a target plot event matched in at least two plot events based on a combination of the multi-modal information in at least two dimensions as the plot event of the video frame; performing semantic annotation on the video frame sequence according to the plot event of the video frame to obtain semantic information of the video.

2. The method of claim 1, wherein, The semantic annotation on the video frame sequence according to the plot event of the video frame to obtain semantic information of the video comprises: determining a start timestamp and an end timestamp of a video segment to which a plurality of continuous video frames with the same plot event belong in the video frame sequence; performing semantic annotation on the video segment according to the plot event, the start timestamp and the end timestamp to obtain semantic information corresponding to the video segment in the video.

3. The method of claim 2, wherein, The determining a target plot event matched in at least two plot events based on a combination of the multi-modal information in at least two dimensions as the plot event of the video frame comprises: obtaining a corresponding relationship, the corresponding relationship being used to represent a corresponding relationship between the combined event of the multi-modal information and a target plot event; obtaining the target plot event according to the corresponding relationship and the combined event of the multi-modal information, and determining the target plot event as the plot event of the video frame.

4. The method of claim 1, wherein, The virtual element information comprises a character identifier corresponding to the first virtual character, and the first information extraction model comprises an element classification model and a positioning and tracking model. The calling a first information extraction model to extract virtual element information from the virtual environment picture comprises: obtaining a first character identifier of the first virtual character participating in the session; calling the element classification model to identify a health value region corresponding to the first virtual character in the virtual environment picture to obtain a second character identifier of the first virtual character corresponding to the health value region; in response to the first character identifier matching the second character identifier, obtaining a first confidence degree corresponding to the second character identifier; The positioning tracking model is called to identify a first virtual character in the virtual environment picture according to the first character identifier, to obtain a third character identifier of the first virtual character displayed in the virtual environment picture, the third character identifier corresponding to a second confidence, the positioning tracking model corresponding to the first virtual character participating in the game in one-to-one manner; The character identifier of the first virtual character is obtained according to the first confidence and the second confidence.

5. The method of claim 1, wherein, The virtual element information includes position information corresponding to a building element in the virtual environment, and the first information extraction model includes a first convolutional neural network; The first information extraction model is called to extract virtual element information from the virtual environment picture, including: A building element feature corresponding to the building element is extracted from the virtual environment picture; The building element feature is matched with a preset building element feature to obtain a first matching feature; The first convolutional neural network is called to classify the first matching feature to obtain the position information corresponding to the building element.

6. The method of claim 1, wherein, The virtual element information includes position information corresponding to a neutral virtual character in the virtual environment, and the first information extraction model includes a second convolutional neural network; The first information extraction model is called to extract virtual element information from the virtual environment picture, including: A character element feature corresponding to the neutral virtual character is extracted from the virtual environment picture; The character element feature is matched with a preset character element feature to obtain a second matching feature; The second convolutional neural network is called to classify the second matching feature to obtain the position information corresponding to the neutral virtual character.

7. The method of claim 1, wherein, The game information includes game end information, and the second information extraction model includes a video frame classification model; The second information extraction model is called to extract the game information from the control, including: The video frame classification model is called to classify the video frame sequence to obtain a video frame corresponding to a game end state, the video frame corresponding to the game end state including the control; Text recognition is performed on the control to obtain a text recognition result; In response to the text recognition result including game results, the game end information is obtained according to the game results.

8. The method of claim 1, wherein, The game information includes game state information between the first virtual characters, and the second information extraction model includes a video frame classification model; The second information extraction model is called to extract the game information from the control, including: The video frame classification model is called to classify the video frame sequence to obtain a game picture frame, the game picture frame including the control; A hitting state in the control is identified to obtain a hitting state result, the hitting state result being an event result corresponding to a hitting event between the first virtual characters; The game state information is obtained according to the hitting state result.

9. The method according to any one of claims 1 to 3, characterized in that, The video frame in the video frame sequence comprises a virtual environment picture area and a live picture area, the live picture area is used to represent a picture area for shooting a live process of an anchor, and the live picture area does not comprise the virtual environment picture area; Before the extracting of the multi-modal information in at least two dimensions from the video frame in the video frame sequence, comprising: determining a boundary between the virtual environment picture area and the live picture area; cutting the live picture area according to the boundary to obtain a video frame containing the virtual environment picture area.

10. The method of claim 9, wherein, The determining of the boundary between the virtual environment picture area and the live picture area comprises: obtaining a video frame sequence after binaryzation processing; converting pixel points contained in a video frame in the video frame sequence after the binaryzation processing to a Huffman space according to a Huffman algorithm; in response to a number of curves intersecting at a same pixel point in the Huffman space being greater than or equal to a number threshold, determining that a straight line corresponding to the pixel point exists in the video frame; determining the boundary between the virtual environment picture area and the live picture area according to the straight line corresponding to the pixel point.

11. The method according to any one of claims 1 to 3, characterized in that, Before the extracting of the multi-modal information in at least two dimensions from the video frame in the video frame sequence, further comprising: calling a backbone network Backbone to perform feature extraction on the video frame in the video frame sequence to obtain a video feature corresponding to the video frame; calling an enhanced super-resolution generative adversarial network ESRGAN to process the video feature to obtain a resolution-enhanced video frame.

12. A video semantic labeling apparatus, characterized by comprising: The apparatus comprises: an obtaining module, configured to obtain a video frame sequence in a video, a video frame in the video frame sequence comprising a virtual environment picture and at least one control, the virtual environment picture comprising a picture of a virtual environment in which a first virtual role is active; an extracting module, configured to extract multi-modal information in at least two dimensions from the video frame in the video frame sequence, the extracting of the multi-modal information in at least two dimensions from the video frame in the video frame sequence comprising: calling a first information extraction model to extract virtual element information from the virtual environment picture, the virtual element information comprising information corresponding to at least one of active elements and static elements in the virtual environment picture; and calling a second information extraction model to extract game information from the control, the game information comprising information corresponding to the first virtual role in the control; a processing module, configured to determine a target plot event in at least two plot events based on a combination of the multi-modal information in at least two dimensions, as a plot event of the video frame; a labeling module, configured to perform semantic labeling on the video frame sequence according to the plot event of the video frame to obtain semantic information of the video.

13. A computer device, comprising: The computer device comprises a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the instruction, the program, the code set or the instruction set being loaded and executed by the processor to implement the video semantic labeling method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The storage medium has stored therein a computer program, and the computer program is loaded and executed by the processor to implement the video semantic labeling method according to any one of claims 1 to 11.

Images