An information processing method, apparatus and computer-readable storage medium

Abstract

Embodiments of the present application disclose an information processing method, apparatus, and computer-readable storage medium. Embodiments of the present application detect video frames to determine target objects in the video frames; determine target object elements corresponding to the target objects in three-dimensional space; determine corresponding event information according to the relationships of the target object elements after clustering; convert the target object elements into corresponding three-dimensional virtual object models, and load the three-dimensional virtual object models into a preset three-dimensional virtual scene; control the three-dimensional virtual object models in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video. In this way, the target object elements in the video frames are identified and the corresponding event information is determined, the target object elements are converted into three-dimensional virtual object models and loaded into the three-dimensional virtual scene, and the three-dimensional virtual object models are controlled according to the event information, so as to realize quickly rendering a two-dimensional video into a three-dimensional video, greatly improving the efficiency of information processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an information processing method, apparatus, and computer-readable storage medium. Background Art

[0002] With the maturity of electronic technology, more and more digital products have emerged. The popularity and prevalence of digital products such as digital cameras, tablet computers, and smart phones have brought great convenience to the general public. In daily life, users can use digital products to shoot or produce their own video works at any time.

[0003] In the prior art, users can edit the shot or produced video works to create personalized movie works. However, video works shot or produced by traditional data products often require the cooperation of a large number of performers or manual creation of anime plots, which is mechanical and cumbersome, resulting in low information processing efficiency. Summary of the Invention

[0004] Embodiments of this application provide an information processing method, apparatus, and computer-readable storage medium, which can improve the efficiency of information processing.

[0005] To solve the above technical problems, embodiments of this application provide the following technical solutions:

[0006] An information processing method, including:

[0007] Detecting a video frame to determine a target object in the video frame;

[0008] Determining a target object element corresponding to the target object in three-dimensional space;

[0009] Determining corresponding event information according to the relationship of the target object elements after clustering;

[0010] Converting the target object element into a corresponding three-dimensional virtual object model, and loading the three-dimensional virtual object model into a preset three-dimensional virtual scene;

[0011] Controlling the three-dimensional virtual object model in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video.

[0012] An information processing apparatus, including:

[0013] A detection unit, configured to detect a video frame to determine a target object in the video frame;

[0014] A first determination unit, configured to determine a target object element corresponding to the target object in three-dimensional space;

[0015] A second determination unit, configured to determine corresponding event information according to the relationship of the target object elements after clustering;

[0016] A conversion unit, configured to convert the target object elements into corresponding three-dimensional virtual object models, and load the three-dimensional virtual object models into a preset three-dimensional virtual scene;

[0017] A control unit, configured to control the three-dimensional virtual object models in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video.

[0018] In some embodiments, the control unit includes:

[0019] A control subunit, configured to control the three-dimensional virtual object models in the three-dimensional virtual scene according to the event information;

[0020] A shooting subunit, configured to shoot the three-dimensional virtual scene through at least one virtual camera device to generate a three-dimensional sub-video;

[0021] A receiving subunit, configured to receive an editing operation of the user on the three-dimensional sub-video to generate a target three-dimensional video.

[0022] In some embodiments, the shooting subunit is configured to:

[0023] Set a virtual shooting movement route for each virtual camera device;

[0024] Move and shoot each virtual camera device along the virtual shooting movement route to obtain a three-dimensional sub-video.

[0025] In some embodiments, the shooting subunit is further configured to:

[0026] Collect images of the user's handheld virtual reality shooting device;

[0027] Generate corresponding virtual shooting devices in the three-dimensional virtual scene according to the virtual reality shooting devices in the images;

[0028] Control the shooting angles and shooting positions of the virtual shooting devices based on the shooting angle information and shooting position information of the virtual reality shooting devices to achieve real-time shooting and generate a three-dimensional sub-video.

[0029] In some embodiments, the first determination unit is configured to:

[0030] Determine the position information of the target object in the video frame;

[0031] Perform three-dimensional extension based on the position information of each target object to generate corresponding target object elements of each target object in three-dimensional space.

[0032] A computer-readable storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to execute the steps in the above information processing method.

[0033] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above information processing method are implemented.

[0034] A computer program product or a computer program includes computer instructions, and the computer instructions are stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the steps in the above information processing method are implemented by the computer.

[0035] In the embodiments of the present application, by detecting a video frame, a target object in the video frame is determined; a target object element corresponding to the target object in a three-dimensional space is determined; corresponding event information is determined according to the relationship after clustering of the target object elements; the target object elements are converted into corresponding three-dimensional virtual object models, and the three-dimensional virtual object models are loaded into a preset three-dimensional virtual scene; and the three-dimensional virtual object models in the three-dimensional virtual scene are controlled according to the event information to generate a target three-dimensional video. In this way, the target object elements in the video frame are recognized and the corresponding event information is determined, the target object elements are converted into three-dimensional virtual object models and loaded into the three-dimensional virtual scene, and the three-dimensional virtual object models are controlled according to the event information. Compared with the prior art that requires a large number of performers to cooperate or manually create anime plots, the present solution can realize quickly rendering a two-dimensional video into a three-dimensional video, greatly improving the efficiency of information processing. Description of the Drawings

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for description in the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 It is a schematic diagram of the scenario of the information processing system provided by the embodiments of the present application;

[0038] Figure 2 It is a schematic flowchart of the information processing method provided by the embodiments of the present application;

[0039] Figure 3 It is another schematic flowchart of the information processing method provided by the embodiments of the present application;

[0040] Figure 4a It is a schematic diagram of the scenario of the information processing method provided by the embodiment of the present application;

[0041] Figure 4b It is another schematic diagram of the scenario of the information processing method provided by the embodiment of the present application;

[0042] Figure 4c It is another schematic diagram of the scenario of the information processing method provided by the embodiment of the present application;

[0043] Figure 5 It is a schematic diagram of the structure of the information processing device provided by the embodiment of the present application;

[0044] Figure 6 It is a schematic diagram of the structure of the server provided by the embodiment of the present application. Specific implementation manners

[0045] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0046] The embodiment of the present application provides an information processing method, device, and computer-readable storage medium.

[0047] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the scenario of the information processing system provided by the embodiment of the present application, including: terminal A, virtual reality device, and server (this information processing system may also include other terminals except terminal A, and the specific number of terminals is not limited here). Terminal A, the virtual reality device, and the server can be connected through a communication network. This communication network can include a wireless network and a wired network, where the wireless network includes one or more combinations of a wireless wide area network, a wireless local area network, a wireless metropolitan area network, and a wireless personal area network. The network includes network entities such as routers and gateways, which are not shown in the figure. Terminal A can perform information interaction with the server through the communication network. For example, terminal A sends game video frames to the server online through a game application.

[0048] The information processing system may include an information processing device, which may be specifically integrated in a server. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. As Figure 1 shown, the server can detect video frames, determine the target objects in the video frames; determine the target object elements corresponding to the target objects in three-dimensional space; determine the corresponding event information according to the relationship of the target object elements after clustering; convert the target object elements into corresponding three-dimensional virtual object models, and load the three-dimensional virtual object models into a preset three-dimensional virtual scene; control the three-dimensional virtual object models in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video.

[0049] In the information processing system, terminal A can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. Terminal A can install various applications required by users, such as game applications. Terminal A can send the video frames of the game in the game application to the server.

[0050] The virtual reality device may include a head-mounted virtual reality device, such as Oculus Rift or HTC Vive. The interactive information of the user can be obtained through the head-mounted virtual reality device and uploaded to the server, and the event information therein can be modified flexibly.

[0051] It should be noted that Figure 1 the scene schematic diagram of the information processing system shown is only an example. The information processing system and the scene described in the embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of the information processing system and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0052] The following will be described in detail respectively.

[0053] In this embodiment, it will be described from the perspective of the information processing device, which may be specifically integrated in a server with a storage unit and installed with a microprocessor and having computing power.

[0054] Please refer to Figure 2 , Figure 2It is a schematic flowchart of the information processing method provided by an embodiment of this application. The information processing method includes:

[0055] In step 101, the video frame is detected to determine the target object in the video frame.

[0056] Among them, the terminal can obtain the running game animation or movie, etc. The game animation or movie is composed of frames of pictures. The terminal uploads the game animation or movie to the server, and the server performs feature detection on the content in the video frames of the game animation or movie, and determines the target object in the video frame and the position information of the target object in the video frame according to the result of the feature detection. The target object can be a person, a tree, a table, a game character, a game defense tower, etc. in the video frame, and no specific limitation is made here.

[0057] In an implementation manner, the step of detecting the video frame to determine the target object in the video frame may include: detecting the virtual object in the video frame through a trained target detection network to determine the target object in the video frame.

[0058] Among them, Artificial Intelligence (AI) is a theory, method, technology and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning and decision-making.

[0059] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0060] Computer Vision Technology (CV) Computer vision is a science that studies how to enable machines to "see". More specifically, it refers to machine vision that uses cameras and computers to replace the human eye for object recognition, detection, and measurement, and further performs graphics processing to make the computer-processed images more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision research related theories and technologies attempt to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc. technologies, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0061] The solution provided in the embodiments of this application relates to technologies such as computer vision technology in artificial intelligence, and is specifically described through the following embodiments:

[0062] Among them, a target detection network (YOLO) can be pre-trained, that is, a large number of training samples of virtual objects that may appear in video frames are pre-collected. The training samples include virtual objects and corresponding label information. By inputting the virtual objects and corresponding label information into the target detection network for training, a target detection network that can recognize virtual objects after training is obtained.

[0063] Furthermore, the trained target detection network is used to detect the virtual objects in the video frame, and the target objects in the video frame and the position information of the target objects in the video frame are determined. For example, each target object can be calibrated in the form of a rectangular box, and the position information of each target object is displayed.

[0064] In step 102, determine the target object elements corresponding to the target objects in three-dimensional space.

[0065] Among them, based on the position information of the target object in the video frame, three-dimensional extension is performed to generate the target object elements corresponding to each target object in three-dimensional space. The target object elements can be three-dimensional instances, and their manifestation forms in three-dimensional space can be spherical nodes, and the spherical nodes represent the temporary three-dimensional instance manifestation forms of each target object in three-dimensional space.

[0066] In some embodiments, the step of determining the target object elements corresponding to the target objects in three-dimensional space may include:

[0067] (1) Determine the position information of the target object in the video frame;

[0068] (2) Perform three-dimensional extension based on the position information of each target object to generate a target object element corresponding to each target object in three-dimensional space.

[0069] Among them, since the position information corresponding to the target object in the video frame has been calibrated, the position information of the target object in the video frame can be directly determined. Based on this, three-dimensional extension is performed into three-dimensional space based on the position information of each target object in the video frame to find the three-dimensional space position data corresponding to the target object element corresponding to each target object in three-dimensional space, and display is performed according to this three-dimensional space position data.

[0070] In step 103, determine the corresponding event information according to the relationship of the target object elements after clustering.

[0071] Among them, clustering operations can be performed on the multiple target object elements corresponding to the target object detected in step 101 to obtain different types of target object sets. Each target object set includes target object set elements of the corresponding type. For example, tables and chairs are clustered into target object set elements of the furniture type, and different users are clustered into target object set elements of the user type, forming target object set elements of different clustering types and multiple target object sets.

[0072] Furthermore, event abstraction is performed according to the clustering type. The basis for this event abstraction is also a predefined abstraction rule. The abstraction rule includes a subject and an object. For example, when there are person clustering, furniture clustering, and food clustering in the image of the video frame, then the subject can be the target object set element of the person clustering, and the object can be the target object set elements of the furniture clustering and the food clustering. According to this abstraction rule, the subject, the object, and the verb between the objects are respectively determined, and then the event information is determined, such as "a person is eating at home" or "a person is processing food", etc.

[0073] In some embodiments, the step of determining the corresponding event information according to the relationship of the target object elements after clustering may include:

[0074] (1) Perform clustering according to the attribute data of the target object elements to obtain different types of target object sets. Each target object set includes target object set elements of the corresponding type;

[0075] (2) Determine the corresponding event elements based on the relationship between the target object set elements. The event elements include event information.

[0076] Among them, clustering can be performed according to the attribute data of the target object element to obtain different types of target object sets. Each target object set includes target object set elements of the corresponding type. The attribute data can be the function or use of the target object element. Clustering is performed according to the attribute data of the target object element by the nearest neighbor clustering method, and the target object set elements with similar functions or uses are clustered to obtain different types of target object sets. For example, clustering is performed by the K-Means clustering algorithm. Each target object element is converted into corresponding vector information. The vector information of some target object elements is randomly selected to initialize the center points of multiple classes, and the vector information of each target object element is classified by calculating the distance from the center points of multiple classes. Then each target object element is classified into the category closest to it. Based on the classification result, the mean value of the vector information of all target object elements in each classification is taken to recalculate the center of the class. In this way, through multiple iterative clustering operations, different types of target object sets are obtained, and the target object elements in each target object set are merged into target object set elements, so that each target object set includes target object set elements of the corresponding type.

[0077] Furthermore, event abstraction is performed according to the clustering type, and the basis for this event abstraction is also a predefined abstraction rule. The abstraction rule includes a subject and an object. For example, when there are clustering types of blue-side soldiers, red-side soldiers, and defense towers in the image of a video frame, then the subject can be determined as the clustering type of the defense tower, and the objects can be the clustering types of blue-side soldiers and red-side soldiers. According to the abstraction rule, the subject, the object, and the verb between the objects are determined respectively, and then the event information can be the event element of "defense tower and minion wave confrontation", and this event information can be understood as script information.

[0078] In step 104, the target object element is converted into a corresponding three-dimensional virtual object model, and the three-dimensional virtual object model is loaded into a preset three-dimensional virtual scene.

[0079] In one embodiment, the application environment of the embodiments of the present application can be Unity3D, and the Unity3D can implement a real-time 3D interactive content creation and operation platform. All creators including game development, art, architecture, automotive design, and film and television can turn their creativity into reality with the help of Unity3D technology.

[0080] Among them, for each target object element, modeling can be pre - carried out in advance to obtain a three - dimensional virtual object model corresponding to each target object element, generate a 3D model library, and import the 3D model library into the Unity3D application environment. The 3D model library can include a detection module and a modeling module. The detection module can be used to, after an input video frame and the recognition of a target object element, based on the modeling module, convert the recognized target object element into a corresponding three - dimensional virtual object model.

[0081] Furthermore, the target object elements can be matched in the 3D model library to find the three - dimensional virtual object model corresponding to each target object element.

[0082] In one embodiment, the subsequent scene configuration information can also be pre - configured to generate a preset three - dimensional virtual scene. For example, some game elements are pre - loaded to generate a preset three - dimensional virtual scene. The three - dimensional virtual object model is loaded in a three - dimensional form into the preset three - dimensional virtual scene to realize the construction of characters and scenes.

[0083] In some embodiments, the steps of converting the target object element into a corresponding three - dimensional virtual object model and loading the three - dimensional virtual object model into a preset three - dimensional virtual scene may include:

[0084] (1) Perform matching according to the attribute data of each target object element to determine the corresponding three - dimensional virtual object model;

[0085] (2) Generate a preset three - dimensional virtual scene;

[0086] (3) Receive a selection instruction for the three - dimensional virtual object model, and the selection instruction carries target three - dimensional position data;

[0087] (4) In response to the selection instruction, load the three - dimensional virtual object model onto the three - dimensional spatial position indicated by the target three - dimensional position data in the preset three - dimensional virtual scene.

[0088] Among them, the attribute data is the function or use of each target object element. The 3D model library can contain the association relationship between the attribute data of the target object element and the corresponding three - dimensional virtual object model. Thus, according to the attribute data of each target object element, matching can be performed in the 3D model library to determine the corresponding three - dimensional virtual object model. The three - dimensional virtual object model can be a virtual character, a virtual defense tower, etc.

[0089] Further, a preset three-dimensional virtual scene is generated according to the event information. The preset three-dimensional virtual scene can be multiple three-dimensional virtual scenes pre-built for the user, such as a forest virtual scene or a home virtual scene, etc. The user can select the required three-dimensional virtual scene for generation. The required three-dimensional virtual scene is combined into the three-dimensional space. The user can drag the three-dimensional virtual object model to the three-dimensional virtual scene to generate a selection instruction, and the selection instruction can carry the three-dimensional position data where the three-dimensional virtual object model to be dragged falls in the three-dimensional virtual scene.

[0090] Thereby, the server responds to the selection instruction and loads the three-dimensional virtual object model onto the three-dimensional space position indicated by the target three-dimensional position data in the preset three-dimensional virtual scene, realizing the rapid setting of the three-dimensional virtual object in the three-dimensional virtual scene.

[0091] In step 105, the three-dimensional virtual object model in the three-dimensional virtual scene is controlled according to the event information to generate a target three-dimensional video.

[0092] Among them, since the event information is script information, that is, the event information can include the order and rules of the event occurrence. Therefore, based on the Unity3D technology, each three-dimensional virtual object in the three-dimensional virtual scene can be controlled according to the event information. For example, controlling the movement of characters, vehicles, animals, or the rising and setting of the sun according to the script information, so that each three-dimensional virtual object automatically performs three-dimensional operation according to the script information, completes the specific script operation, generates the target three-dimensional video, and realizes the rapid conversion of the two-dimensional video into the target three-dimensional video for script operation, greatly improving the information processing efficiency.

[0093] In some embodiments, the step of generating the target three-dimensional video includes:

[0094] (1) Shooting the three-dimensional virtual scene through at least one virtual camera device to generate a three-dimensional sub-video;

[0095] (2) Receiving the user's editing operation on the three-dimensional sub-video to generate a target three-dimensional video.

[0096] Among them, at least one virtual camera device can be set. The virtual camera device can be a virtual camera in the three-dimensional virtual scene. The user can set parameters such as the focal length, field of view angle, and resolution of the virtual camera. The three-dimensional virtual scene and the three-dimensional virtual object model therein are shot in real time through the virtual camera device to generate a three-dimensional sub-video, and multiple three-dimensional sub-videos can be included.

[0097] Further, the user can perform editing operations on the three-dimensional sub-video, such as cutting, fusion, etc., to generate a personalized target three-dimensional video.

[0098] In some embodiments, the step of capturing the three-dimensional virtual scene by at least one virtual camera device to generate a three-dimensional sub-video includes:

[0099] (1.1) Collecting an image of a user holding a virtual reality shooting device;

[0100] (1.2) Generating a corresponding virtual shooting device in the three-dimensional virtual scene based on the virtual reality shooting device in the image;

[0101] (1.3) Controlling the shooting angle and shooting position of the virtual shooting device based on the shooting angle information and shooting position information of the virtual reality shooting device to achieve real-time shooting and generate a three-dimensional sub-video.

[0102] Among them, the user can wear a virtual reality shooting device, and the server can collect an image of the user holding the virtual reality shooting device in real time through a camera or an infrared sensor, and generate a corresponding virtual shooting device in the three-dimensional virtual scene based on the state of the virtual reality shooting device held by the user in the image. That is, it can be realized that a virtual character holds a virtual shooting device and enters the three-dimensional virtual scene. The shooting angle and shooting position of the virtual shooting device in the three-dimensional virtual scene are updated in real time based on the shooting angle information and shooting position information of the virtual reality shooting device, so as to realize that a real person controls the virtual reality shooting device in the three-dimensional virtual scene through the virtual reality shooting device to perform three-dimensional shooting, and can perform operations such as capturing and following a specific virtual character, and generate a real-time three-dimensional sub-video.

[0103] As can be seen from the above, in the embodiment of the present application, by detecting a video frame, the target object in the video frame and the target object element corresponding to the target object in the three-dimensional space are determined; the corresponding event information is determined according to the relationship after clustering of the target object elements; the target object elements are converted into corresponding three-dimensional virtual object models, and the three-dimensional virtual object models are loaded into a preset three-dimensional virtual scene; the three-dimensional virtual object models in the three-dimensional virtual scene are controlled according to the event information to generate a target three-dimensional video. In this way, the target object elements in the video frame are recognized and the corresponding event information is determined, the target object elements are converted into three-dimensional virtual object models and loaded into the three-dimensional virtual scene, and the three-dimensional virtual object models are controlled according to the event information. Compared with the prior art that requires a large number of performers to cooperate or manually create anime plots, this solution can realize rapid rendering of two-dimensional videos into three-dimensional videos, greatly improving the efficiency of information processing.

[0104] Combined with the method described in the above embodiments, the following will give further detailed examples.

[0105] In this embodiment, it will be described by taking the specific integration of the information processing device in the server as an example, and the following description will be specifically referred to.

[0106] Please refer to Figure 3 , Figure 3 , which is another flowchart of the information processing method provided by the embodiment of the present application. The method process may include:

[0107] In step 201, the server detects virtual objects in the video frame through the trained target detection network to determine the target objects in the video frame.

[0108] Among them, the server can obtain the game animation running on the terminal. The game animation is composed of frames of images. The server detects virtual objects in the video frame through the trained target detection network to determine the target objects in the video frame.

[0109] Please also refer to Figure 4a As shown, the server detects virtual objects in video frame 10 through the trained target detection network to determine target objects 11, 12, 13, and 14 in video frame 10.

[0110] In step 202, the server determines the position information of the target object in the video frame, and performs three-dimensional extension based on the position information of each target object to generate corresponding target object elements of each target object in the three-dimensional space.

[0111] Among them, please also refer to Figure 4a As shown, the server determines the position information of target objects 11, 12, 13, and 14 in video frame 10, and performs three-dimensional extension based on the position information of target objects 11, 12, 13, and 14 to generate corresponding target object elements A, B, C, and D of target objects 11, 12, 13, and 14 in the three-dimensional space. The target object element can be represented as a circular node in the three-dimensional space.

[0112] In step 203, the server clusters according to the attribute data of the target object elements to obtain different types of target object sets, and determines corresponding event elements based on the relationships between the target object set elements.

[0113] Among them, please also refer to Figure 4aAs shown, the server clusters based on the attribute data of target object elements A, B, C, and D. It clusters target object elements A and B with similar attribute data, and clusters target object elements C and D with similar attribute data, obtaining different types of target object sets 1 and 2. The target object set 1 includes target object set element A of the corresponding type, and the target object set 2 includes target object set element B of the corresponding type. Assuming that the type of target object set element A is a blue-side soldier and the type of target object set element B is a red-side soldier, then an event element of the event type information of the frontline confrontation is determined according to the relationship between target object set element A and target object set element B.

[0114] In step 204, the server determines the three-dimensional space position data corresponding to the target object set element and the three-dimensional space position data corresponding to the event element according to the event information. Based on the three-dimensional space position data corresponding to the target object set element and the three-dimensional space position data corresponding to the event element, each target object set element and the event element are loaded with positions.

[0115] Among them, please continue to refer to Figure 4a As shown, the server determines the three-dimensional space position data corresponding to target object element A, target object element B, and the event element in the three-dimensional space according to the event information of the frontline confrontation event type. Based on this three-dimensional space position data, target object element A, target object element B, and the event element are respectively loaded with positions in the three-dimensional space, and target object element A, target object element B, and the event element are placed in the three-dimensional space in a hierarchical structure form, that is, the elements are expressed layer by layer in a structured form, showing the relationship information between the elements.

[0116] In step 205, the server connects the lines according to the position relationship among the target object element, the target object set element, and the event element in the three-dimensional space, generates a three-dimensional structure element set and displays it.

[0117] Among them, please continue to refer to Figure 4a As shown, the server connects the lines according to the logical relationship among target object element A, target object element B, target object element C, target object element D, target object set element A, target object set element B, and the event element displayed in the three-dimensional space. For example, target object element A and target object element B of the similar clustering type are connected to target object set element A, target object element C and target object element D of the similar clustering type are connected to target object set element B, and target object combination element A and target object set element B are connected to the event element, generating a three-dimensional structure element set.

[0118] In step 206, the server receives the modification operation of the modification object in the three-dimensional structure element set by the user through the virtual reality device, and updates the event elements in the three-dimensional structure element set after the modification operation.

[0119] Among them, please refer to Figure 4b As shown, the real human (i.e., the user) 21 can perform a modification operation on the modification object in the three-dimensional structure element set by wearing a virtual reality device. The modification object can include at least one of the target object element or the connection line. For example, adding a target object element, deleting a target object element, realizing rapid personalized modification of the target object element or the connection line in the three-dimensional space in the virtual reality scenario, and enhancing the diversity of information processing.

[0120] In an embodiment, the connection line between the target object element and the target object set element can also be modified, or a connection line can be established between the newly added target object element and the target object set element to be classified, so as to realize the modification of the event element, that is, the script. For example, adding a target object element F, which is a virtual defense tower, and determining the corresponding target object set element C based on the virtual defense tower. The clustering type of the target object set element C is a defense tower. In this way, the event elements in the three-dimensional structure element set after the modification operation are updated, and the event information of the event element is modified from "lane confrontation" to "defense tower lane confrontation", realizing the modification of the script.

[0121] In step 207, the server performs matching according to the attribute data of each target object element, determines the corresponding three-dimensional virtual object model, generates a preset three-dimensional virtual scene, and receives a selection instruction for the three-dimensional virtual object model.

[0122] Among them, please refer to Figure 4c As shown, the server performs matching in the 3D model library according to the attribute data of each target object element, and finds the three-dimensional virtual object model with the same attribute data. For example, determining the three-dimensional virtual object model corresponding to a person, the three-dimensional virtual object model corresponding to a table, and the three-dimensional virtual object model corresponding to a tree, generating a preset three-dimensional virtual scene, and receiving a selection instruction for the three-dimensional virtual object model corresponding to a person, the three-dimensional virtual object model corresponding to a table, and the three-dimensional virtual object model corresponding to a tree. The selection instruction carries the target three-dimensional position data for placing the three-dimensional virtual object model in the preset three-dimensional virtual scene.

[0123] In step 208, the server responds to the selection instruction and loads the three-dimensional virtual object model to the three-dimensional space position indicated by the target three-dimensional position data in the preset three-dimensional virtual scene.

[0124] Among them, please continue to refer toFigure 4c As shown, in response to the selection instruction, the server loads the 3D virtual object model corresponding to the character, the 3D virtual object model corresponding to the table, and the 3D virtual object model corresponding to the tree to the 3D space position indicated by the target 3D position data in the preset 3D virtual scene.

[0125] In step 209, the server controls the 3D virtual object models in the 3D virtual scene according to the event information, and sets the virtual shooting movement routes of each virtual camera device.

[0126] Among them, please continue to refer to Figure 4c As shown, the server can control the 3D virtual object model corresponding to the character, the 3D virtual object model corresponding to the table, and the 3D virtual object model corresponding to the tree in the 3D virtual scene according to the event information of the life scene. For example, it controls the 3D virtual object model corresponding to the character to perform daily life activities in the 3D virtual scene, simulating a 3D life scene.

[0127] Furthermore, the number of the virtual camera devices can be arbitrary. For example, Figure 4c As shown, assuming there are 4 virtual camera devices, the creator can set the starting position of each virtual camera device in the 3D space and the virtual shooting movement route, and the virtual shooting movement route can be zoom out, zoom in, pan or tilt.

[0128] In step 210, the server moves and shoots each virtual camera device along the virtual shooting movement route to obtain a 3D sub-video, receives the editing operation of the user on the 3D sub-video, and generates a target 3D video.

[0129] Among them, when the script starts, that is, after the server controls the 3D virtual object models in the 3D virtual scene according to the event information, the 3D virtual object models move according to the event information. The server can control each virtual camera device to move and shoot along the virtual shooting movement route to obtain four 3D sub-videos obtained by each virtual camera device moving and shooting on the virtual shooting movement route.

[0130] Furthermore, the user can edit the videos in the four 3D sub-videos obtained by shooting on the server. For example, cut the required video segments according to the requirements, and fuse the cut video segments according to the personalization to generate a personalized target 3D video, and the target 3D video contains a collection of 3D videos shot from multiple shooting angles.

[0131] As can be seen from the above, in the embodiment of the present application, by detecting a video frame, a target object in the video frame and a target object element corresponding to the target object in three-dimensional space are determined; event information corresponding to the relationship after clustering of the target object elements is determined; the target object elements are converted into corresponding three-dimensional virtual object models, and the three-dimensional virtual object models are loaded into a preset three-dimensional virtual scene; the three-dimensional virtual object models in the three-dimensional virtual scene are controlled according to the event information to generate a target three-dimensional video. In this way, the target object elements in the video frame are recognized and the corresponding event information is determined, the target object elements are converted into three-dimensional virtual object models and loaded into the three-dimensional virtual scene, and the three-dimensional virtual object models are controlled according to the event information. Compared with the prior art that requires a large number of performers to cooperate or manually create anime plots, this solution can realize the rapid rendering of two-dimensional videos into three-dimensional videos, greatly improving the efficiency of information processing.

[0132] Furthermore, in the embodiment of the present application, it is also possible to enable a user wearing a virtual reality device to design and modify the event information of event elements, providing a flexible event design method and further improving the efficiency of information processing.

[0133] To facilitate better implementation of the information processing method provided by the embodiment of the present application, the embodiment of the present application also provides a device based on the above information processing method. The meanings of the nouns are the same as those in the above information processing method, and the specific implementation details can refer to the description in the method embodiment.

[0134] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of the information processing device provided by the embodiment of the present application. The information processing device may include a detection unit 301, a first determination unit 302, a second determination unit 303, a conversion unit 304, a control unit 305, etc.

[0135] The detection unit 301 is configured to detect a video frame and determine the target object in the video frame.

[0136] The first determination unit 302 is configured to determine the target object element corresponding to the target object in three-dimensional space.

[0137] In some embodiments, the first determination unit 302 is configured to:

[0138] Determine the position information of the target object in the video frame;

[0139] Perform three-dimensional extension based on the position information of each target object to generate a target object element corresponding to each target object in three-dimensional space.

[0140] The second determination unit 303 is configured to determine corresponding event information according to the relationship of the target object elements after clustering.

[0141] In some embodiments, the second determination unit 303 includes:

[0142] A clustering subunit, configured to perform clustering on the attribute data of the target object elements to obtain different types of target object sets, and each target object set includes target object set elements of the corresponding type;

[0143] A determination subunit, configured to determine corresponding event elements based on the relationship between the target object set elements, and the event elements include event information.

[0144] In some embodiments, the apparatus further includes:

[0145] A third determination unit, configured to determine the three-dimensional spatial position data corresponding to the target object set elements and the three-dimensional spatial position data corresponding to the event elements according to the event information;

[0146] A loading unit, configured to perform position loading on each target object set element and event element based on the three-dimensional spatial position data corresponding to the target object set elements and the three-dimensional spatial position data corresponding to the event elements;

[0147] A generating unit, configured to connect connection lines according to the position relationship among the target object elements, target object set elements, and event elements in the three-dimensional space, generate a three-dimensional structure element set and display it.

[0148] In some embodiments, the apparatus further includes:

[0149] A modification unit, configured to receive a modification operation of a modification object in the three-dimensional structure element set by a user through a virtual reality device, where the modification object includes at least one of a target object element or a connection line;

[0150] An update unit, configured to update the event elements in the three-dimensional structure element set after the modification operation.

[0151] A conversion unit 304, configured to convert the target object elements into corresponding three-dimensional virtual object models, and load the three-dimensional virtual object models into a preset three-dimensional virtual scene.

[0152] In some embodiments, the conversion unit 304 is configured to:

[0153] Match according to the attribute data of each target object element to determine the corresponding three-dimensional virtual object model;

[0154] Generate a preset three-dimensional virtual scene;

[0155] Receive a selection instruction for the three-dimensional virtual object model, where the selection instruction carries target three-dimensional position data;

[0156] In response to the selection instruction, load the three-dimensional virtual object model to the three-dimensional spatial position indicated by the target three-dimensional position data in the preset three-dimensional virtual scene.

[0157] The control unit 305 is configured to control the three-dimensional virtual object model in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video.

[0158] In some embodiments, the control unit 304 includes:

[0159] A control subunit for controlling the three-dimensional virtual object model in the three-dimensional virtual scene according to the event information;

[0160] A shooting subunit for shooting the three-dimensional virtual scene through at least one virtual camera device to generate a three-dimensional sub-video;

[0161] A receiving subunit for receiving an editing operation of the user on the three-dimensional sub-video to generate a target three-dimensional video.

[0162] In some embodiments, the shooting subunit is configured to:

[0163] Set the virtual shooting movement route of each virtual camera device;

[0164] Move and shoot each virtual camera device along the virtual shooting movement route to obtain a three-dimensional sub-video.

[0165] In some embodiments, the shooting subunit is further configured to:

[0166] Collect images of the user's hand-held virtual reality shooting device;

[0167] Generate a corresponding virtual shooting device in the three-dimensional virtual scene according to the virtual reality shooting device in the image;

[0168] Control the shooting angle and shooting position of the virtual shooting device based on the shooting angle information and shooting position information of the virtual reality shooting device to achieve real-time shooting and generate a three-dimensional sub-video.

[0169] For the specific implementation of each of the above units, reference may be made to the previous embodiments, which will not be elaborated here.

[0170] As can be seen from the above, in the embodiment of the present application, the detection unit 301 detects a video frame to determine a target object in the video frame; the first determination unit 302 determines a target object element corresponding to the target object in the three-dimensional space; the second determination unit 303 determines corresponding event information according to the relationship of the target object elements after clustering; the conversion unit 304 converts the target object elements into corresponding three-dimensional virtual object models, and loads the three-dimensional virtual object models into a preset three-dimensional virtual scene; the control unit 305 controls the three-dimensional virtual object models in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video. In this way, the target object elements in the video frame are recognized and the corresponding event information is determined, the target object elements are converted into three-dimensional virtual object models and loaded into the three-dimensional virtual scene, and the three-dimensional virtual object models are controlled according to the event information. Compared with the prior art that requires a large number of performers to cooperate or manually create anime plots, this solution can quickly render a two-dimensional video into a three-dimensional video, greatly improving the efficiency of information processing.

[0171] The embodiment of the present application also provides a computer device, which can be a server, such as Figure 6 shown, which shows a schematic structural diagram of the server involved in the embodiment of the present application. Specifically:

[0172] This computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art can understand that Figure 6 the computer device structure shown in does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or arrange different components. Among them:

[0173] The processor 401 is the control center of this computer device, connecting various parts of the entire computer device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, it executes various functions of the computer device and processes data, thereby performing overall detection of the computer device. Optionally, the processor 401 may include one or more processing cores; optionally, the processor 401 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above modem processor may not be integrated into the processor 401.

[0174] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the server, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 402 can also include a memory controller to provide the processor 401 with access to the memory 402.

[0175] The computer device further includes a power supply 403 for powering each component. Optionally, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0176] The computer device may further include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0177] Although not shown, the computer device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402 to implement the various method steps provided in the foregoing embodiments, as follows:

[0178] Detect a video frame to determine a target object in the video frame; determine a target object element corresponding to the target object in three-dimensional space; determine corresponding event information according to the relationship of the target object elements after clustering; convert the target object elements into corresponding three-dimensional virtual object models, and load the three-dimensional virtual object models into a preset three-dimensional virtual scene; control the three-dimensional virtual object models in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video.

[0179] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not elaborated in a certain embodiment, reference may be made to the detailed description of the information processing method above, and details are not repeated here.

[0180] As can be seen from the above, the computer device according to the embodiment of the present application can detect a video frame to determine a target object in the video frame; determine a target object element corresponding to the target object in three-dimensional space; determine corresponding event information according to the relationship of the target object elements after clustering; convert the target object elements into corresponding three-dimensional virtual object models, and load the three-dimensional virtual object models into a preset three-dimensional virtual scene; control the three-dimensional virtual object models in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video. In this way, the target object elements in the video frame are identified and the corresponding event information is determined, the target object elements are converted into three-dimensional virtual object models and loaded into the three-dimensional virtual scene, and the three-dimensional virtual object models are controlled according to the event information. Compared with the prior art solutions that require a large number of performers to cooperate or manually create anime plots, this solution can quickly render a two-dimensional video into a three-dimensional video, greatly improving the efficiency of information processing.

[0181] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0182] Therefore, the embodiment of the present application provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the information processing methods provided by the embodiment of the present application. For example, the instructions can execute the following steps:

[0183] Detect a video frame to determine the target object in the video frame; determine the target object element corresponding to the target object in three-dimensional space; determine the corresponding event information according to the relationship of the target object elements after clustering; convert the target object elements into corresponding three-dimensional virtual object models, and load the three-dimensional virtual object models into a preset three-dimensional virtual scene; control the three-dimensional virtual object models in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video.

[0184] According to one aspect of the present application, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various alternative implementations provided in the above embodiments.

[0185] For the specific implementation of each of the above operations, reference may be made to the previous embodiments and will not be elaborated here.

[0186] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0187] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the information processing methods provided in the embodiments of the present application, the beneficial effects achievable by any of the information processing methods provided in the embodiments of the present application can be achieved. For details, reference may be made to the previous embodiments and will not be elaborated here.

[0188] The above has introduced in detail an information processing method, apparatus and computer-readable storage medium provided by the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An information processing method, characterized in that, Including: Detect a video frame to determine a target object in the video frame; Determine a target object element corresponding to the target object in three-dimensional space; Determine corresponding event information according to the relationship of the target object elements after clustering; Wherein, it includes: clustering according to the attribute data of the target object elements to obtain different types of target object sets, each target object set includes target object set elements of the corresponding type; determining corresponding event elements based on the relationship between the target object set elements, and the event elements include event information; Convert the target object element into a corresponding three-dimensional virtual object model, and load the three-dimensional virtual object model into a preset three-dimensional virtual scene; Control the three-dimensional virtual object model in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video; Determine the three-dimensional space position data corresponding to the target object set elements and the three-dimensional space position data corresponding to the event elements according to the event information; Based on the three-dimensional space position data corresponding to the target object set elements and the three-dimensional space position data corresponding to the event elements, perform position loading on each target object set element and event element; Connect the lines according to the position relationship among the target object elements, target object set elements and event elements in three-dimensional space, generate a three-dimensional structure element set and display it.

2. The information processing method according to claim 1, wherein The method further includes: Receiving a modification operation of a modified object in the three-dimensional structure element set by the user through a virtual reality device, and the modified object includes at least one of a target object element or a connection line; Update the event elements in the three-dimensional structure element set after the modification operation.

3. The information processing method according to any one of claims 1 to 2, characterized in that The step of converting the target object element into a corresponding three-dimensional virtual object model and loading the three-dimensional virtual object model into a preset three-dimensional virtual scene includes: Perform matching according to the attribute data of each target object element to determine a corresponding three-dimensional virtual object model; Generate a preset three-dimensional virtual scene; Receive a selection instruction for the three-dimensional virtual object model, and the selection instruction carries target three-dimensional position data; Respond to the selection instruction and load the three-dimensional virtual object model to the three-dimensional space position indicated by the target three-dimensional position data in the preset three-dimensional virtual scene.

4. The information processing method according to any one of claims 1 to 2, characterized in that, The step of generating the target three-dimensional video includes: Shoot the three-dimensional virtual scene through at least one virtual camera device to generate a three-dimensional sub-video; Receive an editing operation of the user on the three-dimensional sub-video to generate a target three-dimensional video.

5. The information processing method according to claim 4, wherein The step of shooting the three-dimensional virtual scene through at least one virtual camera device to generate a three-dimensional sub-video includes: Set the virtual shooting movement route of each virtual camera device; Move and shoot each virtual camera device along the virtual shooting movement route to obtain a three-dimensional sub-video.

6. The information processing method according to claim 4, wherein The step of shooting the three-dimensional virtual scene through at least one virtual camera device to generate a three-dimensional sub-video includes: Collect the image of the user's hand-held virtual reality shooting device; Generate a corresponding virtual shooting device in the three-dimensional virtual scene according to the virtual reality shooting device in the image; Control the shooting angle and shooting position of the virtual shooting device based on the shooting angle information and shooting position information of the virtual reality shooting device to achieve real-time shooting and generate a three-dimensional sub-video.

7. The information processing method according to any one of claims 1 to 2, characterized in that The step of determining the target object element corresponding to the target object in the three-dimensional space includes: Determine the position information of the target object in the video frame; Perform three-dimensional extension based on the position information of each target object to generate a target object element corresponding to each target object in the three-dimensional space.

8. An information processing apparatus, characterized in that, It includes: A detection unit for detecting a video frame and determining the target object in the video frame; A first determination unit for determining the target object element corresponding to the target object in the three-dimensional space; A second determination unit for determining the corresponding event information according to the relationship of the target object elements after clustering; wherein, the second determination unit includes: a clustering subunit for clustering according to the attribute data of the target object elements to obtain different types of target object sets, and each target object set includes target object set elements of the corresponding type; a determination subunit for determining the corresponding event elements based on the relationship between the target object set elements, and the event elements include event information; A conversion unit for converting the target object element into a corresponding three-dimensional virtual object model and loading the three-dimensional virtual object model into a preset three-dimensional virtual scene; A control unit for controlling the three-dimensional virtual object model in the three-dimensional virtual scene according to the event information to generate a target three-dimensional video; A third determination unit for determining the three-dimensional space position data corresponding to the target object set element and the three-dimensional space position data corresponding to the event element according to the event information; A loading unit for performing position loading on each target object set element and event element based on the three-dimensional space position data corresponding to the target object set element and the three-dimensional space position data corresponding to the event element; A generation unit for connecting connection lines according to the position relationship among the target object elements, target object set elements, and event elements in the three-dimensional space, generating a three-dimensional structure element set and displaying it.

9. The information processing apparatus according to claim 8, wherein The device further includes: A modification unit for receiving a modification operation of a modification object in the three-dimensional structure element set by the user through a virtual reality device, and the modification object includes at least one of a target object element or a connection line; An update unit for updating the event elements in the three-dimensional structure element set after the modification operation.

10. The information processing apparatus according to any one of claims 8 to 9, characterized in that, The conversion unit is used for: Matching according to the attribute data of each target object element to determine the corresponding three-dimensional virtual object model; Generate a preset three-dimensional virtual scene; Receive a selection instruction for the three-dimensional virtual object model, and the selection instruction carries target three-dimensional position data; Respond to the selection instruction and load the three-dimensional virtual object model to the three-dimensional space position indicated by the target three-dimensional position data in the preset three-dimensional virtual scene.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores multiple instructions, and the instructions are suitable to be loaded by a processor to execute the steps in the information processing method according to any one of claims 1 to 7.

Images