A video processing method and related device

Abstract

The embodiment of the present application discloses a video processing method and related devices, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The game environment image of the target game can be displayed by setting the display area. Since the game environment image displayed by the display area serves as the background of the entity object, the entity object in the video data is like being in the scene of the target game, which plays a role of telepresence. By identifying the appearance parameters of the entity object in the video data, the target game object can be determined from multiple game objects of the target game based on the appearance parameters, and the virtual props corresponding to the target game object can be rendered on the entity object in the video data to obtain the target video. Since the target game object is determined based on the appearance parameters, it is correlated with the entity object in appearance, which plays a role in the appearance of the entity object playing the target game object in the target game scene.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a video processing method and related devices. Background Art

[0002] For gaming applications, in addition to providing gaming services to users through the game itself, they can also provide users with an immersive experience by building a gaming environment offline.

[0003] The current offline experience mainly uses physical game props (such as costumes, holding props, accessories, etc.) to provide users with game role-playing experience services. Frequent putting on and taking off props is not only troublesome, but also difficult to bring immersion to users.

[0004] It can be seen that how to provide high-quality offline gaming immersive experience is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a video processing method and related devices, which generate a target video so that a physical object can play the role of a target game object in a target game scene, which not only brings a better immersive experience but also makes it interesting.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a video processing method, the method comprising:

[0008] Acquire video data collected for a display area, where the display area is used to display a game environment image of a target game, and where there is a physical object in the display area;

[0009] Determining shape parameters of the physical object in the video data;

[0010] Determining a target game object among a plurality of game objects of the target game according to the appearance parameter;

[0011] The virtual props corresponding to the target game object are rendered on the physical object in the video data to obtain the target video.

[0012] On the other hand, an embodiment of the present application provides a video processing device, the device comprising an acquisition unit, a determination unit and a rendering unit:

[0013] The acquisition unit is used to acquire video data collected for a display area, where the display area is used to display a game environment image of a target game, and where there is a physical object in the display area;

[0014] The determining unit is used to determine the appearance parameters of the entity object in the video data;

[0015] The determining unit is further used to determine a target game object among a plurality of game objects of the target game according to the appearance parameter;

[0016] The rendering unit is used to render the virtual props corresponding to the target game object on the physical object in the video data to obtain the target video.

[0017] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:

[0018] The memory is used to store program codes and transmit the program codes to the processor;

[0019] The processor is used to execute the method described in the above aspects according to the instructions in the program code.

[0020] On the other hand, an embodiment of the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the above aspects.

[0021] On the other hand, an embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method described in the above aspects.

[0022] It can be seen from the above technical solution that the game environment image of the target game can be displayed through the set display area. When the physical object enters the display area, video data can be collected for the display area. Since the game environment image displayed in the display area serves as the background of the physical object, the physical object in the video data is like being in the scene of the target game, which plays a role of telepresence. By identifying the appearance parameters of the physical object in the video data, the target game object can be determined from multiple game objects of the target game based on the appearance parameters, and the virtual props corresponding to the target game object can be rendered on the physical object in the video data to obtain the target video. Since the target game object is determined based on the appearance parameters, it has a correlation with the physical object in appearance, which plays the role of the physical object playing the role of the target game object in the target game scene, which not only brings a better immersive experience, but also has fun. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of a video processing scenario provided in an embodiment of the present application;

[0025] Figure 2 A method flow chart of a video processing method provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of key points of joints of a human body provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of special effects rendering for different human bodies provided in an embodiment of the present application;

[0028] Figure 5a A schematic diagram of a video processing flow provided in an embodiment of the present application;

[0029] Figure 5b A schematic diagram of an offline experience scenario provided in an embodiment of the present application;

[0030] Figure 6a One of the display contents of an operation console provided in an embodiment of the present application;

[0031] Figure 6b The second display content of an operation console provided in an embodiment of the present application;

[0032] Figure 6c The third display content of an operation console provided in an embodiment of the present application;

[0033] Figure 7 A game scene diagram displayed on three screens provided in an embodiment of the present application;

[0034] Figure 8 A rendering of a virtual prop provided in an embodiment of the present application;

[0035] Figure 9a A schematic diagram of a single-person experience process provided in an embodiment of the present application;

[0036] Figure 9b A schematic diagram of a multi-person experience process provided in an embodiment of the present application;

[0037] Fig.10A device structure diagram of a video processing device provided in an embodiment of the present application;

[0038] Fig.11 A structural diagram of a terminal device provided in an embodiment of the present application;

[0039] Fig.12 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0041] Building a high-quality offline game immersive experience is an important way to enhance user stickiness and spread the game audience. However, related technologies mainly use physical game props, and the complex operation makes it difficult to achieve sufficient immersion.

[0042] To this end, an embodiment of the present application provides a video processing method, which generates a target video so that a physical object can play the role of a target game object in a target game scene, which not only brings a better immersive experience but also makes it interesting.

[0043] The video processing method provided in the embodiment of the present application can be implemented by a computer device, which can be a terminal device or a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0044] The embodiments of the present application can be applied to artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0045] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0046] The present application relates to computer vision technology (Computer Vision, CV), which is a science that studies how to make machines "see". To put it more concretely, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, etc. For example, the present application can identify the actions of physical objects in video data through computer vision technology.

[0047] It is understandable that in the specific implementation of the present application, the appearance parameters and other data involved may be related to user information. When the above embodiments of the present application are applied to specific products or technologies, the user's separate permission or consent is required, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0048] Figure 1 A schematic diagram of a video processing scenario provided in an embodiment of the present application, in Figure 1 In the description, the terminal device 100 is taken as the aforementioned computer device for example. The camera 200 is used to collect video data of the display area 300, and the terminal device 100 obtains the collected video data through the camera 200. The display area 300 displays a game environment image of the target game.

[0049] When the camera 200 is collecting video data, user a is in the display area 300 as an entity object. Since the game environment image displayed in the display area serves as the background of user a, user a in the video data seems to be in the scene of the target game, which plays a role of presence.

[0050] The terminal device 100 can determine the appearance parameters of user a based on the video data, such as a height parameter of 1.8 meters. According to the height parameter, a target game object matching user a is determined from multiple game objects of the target game, such as a tall male fighter, and a virtual prop corresponding to the male fighter, such as a boxing glove, is rendered on user a in the video data to obtain a target video, as shown in the display interface 20 of the terminal device 100.

[0051] Since the male fighter determined based on the height parameters is similar in height to user a, user a rendered with gloves in the target video will be more similar to the male fighter in the target game, giving user a the feeling of playing the male fighter in the target game scene, which not only brings a better immersive experience, but also is interesting, attracting users to try which game character they can play.

[0052] Figure 2 A method flow chart of a video processing method provided in an embodiment of the present application is provided. In this embodiment, a terminal device is used as the aforementioned computer device as an example for explanation. The method includes:

[0053] S201: Acquire video data collected for a display area.

[0054] Among them, the display area is used to display the game environment image of the target game. When collecting video data, there is a physical object in the display area. In the video collection direction, the physical object is between the collection device and the display area. Therefore, the physical object in the collected video data is in the background of the game environment image, which can bring a sense of presence in the target game scene.

[0055] The game environment image can be a static image or a dynamic image, and this application does not limit this.

[0056] The entity object may be any actual object with entity, such as a human body, an animal, or any object, etc., and this application does not limit this.

[0057] In order to improve the sense of presence that can be embodied by the display area, in a possible implementation manner, the display area is a three-sided screen, and the three display screens of the three-sided screen are arranged perpendicular to each other.

[0058] The three display screens of the three-sided screen can form a rectangular corner-shaped space by being arranged vertically. The three-dimensional display area can bring a three-dimensional display effect and enhance the sense of reality. When the physical object is in the display area of ​​the three-sided screen, there are realistic game environment images on the left and right sides and above or below, which enhances the sense of presence in the target game scene.

[0059] The present application does not limit the number and type of entity objects in the video data. There may be multiple entity objects or one entity object. The entity objects may all be of the same type, such as human bodies, or may include different types, such as human bodies and animals.

[0060] S202: Determine the appearance parameters of the entity object in the video data.

[0061] The appearance parameters are used to identify the appearance characteristics of the entity object, which can be reflected by body shape or appearance. Therefore, the terminal device can effectively describe the physical characteristics of the entity object through the appearance characteristics, which is convenient for subsequent identification of the target game object that matches the physical object in appearance.

[0062] When the entity object is a human body, S202 includes: determining at least one of facial features, height parameters, and body shape parameters of the human body in the video data as the appearance parameter.

[0063] It should be noted that although the determined appearance parameters include at least one of the facial features, height parameters, and body shape parameters of the human body, these appearance parameters are only used for temporary game object matching. After the matching is completed, the appearance parameters will not be saved or used for other purposes. In addition, the determined appearance parameters are relatively coarse-grained, as long as they can reflect the appearance characteristics, and will not involve user privacy information.

[0064] When the entity object is an animal, the appearance parameters can also identify the physical characteristics, species characteristics, etc. of the animal.

[0065] When the entity object is an object, the shape parameters can also identify the structural characteristics and physical form of the object.

[0066] S203: Determine a target game object from among multiple game objects of the target game according to the appearance parameters.

[0067] Multiple game objects in the target game have corresponding virtual images, which can be embodied by virtual models, for example. Based on the virtual image, the image characteristics of the game object can be determined, and the appearance parameters can reflect the appearance characteristics of the physical object, so the terminal device can determine the target game object from multiple game objects based on the appearance parameters.

[0068] The determined target game object has a correlation in appearance with the entity object in the video data. The correlation in appearance can be reflected in a relatively consistent appearance or a greatly different appearance. A consistent appearance can make the entity object in the target video look like it is playing the target game object, and a greatly different appearance can make the entity object in the target video have a sense of contrast after being rendered as a virtual prop, bringing fun to the offline experience.

[0069] The present application does not limit the type of game objects, which may be game characters of various forms and types, and the specific image and type of the game objects may be related to the target game.

[0070] In a possible implementation, S203 includes: determining, according to the appearance parameter, a target game object that matches the entity object among a plurality of game objects of the target game.

[0071] In this way, the target game object matched is similar in appearance to the entity object. For example, the target game object matched to a tall entity object can be a tall game character, the target game object matched to a petite entity object can be a petite game character, and the target game object matched to a puppy as an entity object can be the puppy virtual image in the target game, etc.

[0072] Since the target game object is similar to the physical object in appearance, the physical object rendered with virtual props in the target video will be more similar to the target game object, giving the physical object the appearance of the target game object in the target game scene.

[0073] S204: Rendering the virtual props corresponding to the target game object on the physical object in the video data to obtain a target video.

[0074] The virtual props corresponding to the target game object can be used to identify the appearance characteristics of the target game object and play a role in distinguishing it from other game objects to a certain extent. The virtual props can be virtual props used to decorate or wear on the target game object in the target game, for example, it can be at least one of virtual accessories, clothing, skin, hair or holding props. The virtual props can also include additional virtual props related to the target game object, such as virtual pets, halo special effects materials, etc., which are not limited in this application.

[0075] Since the appearance of the matched target game object is similar to that of the physical object, the terminal device renders the virtual props corresponding to the target game object onto the physical object in the video data, so that in the target video, the physical object seems to be playing the target game object, giving the appearance that the physical object is playing the target game object in the target game scene. The playback of the target video can bring a better immersive experience and can also attract more participants to try to determine which target game object they can match, making the offline experience more interesting.

[0076] As mentioned above, the target game object determined by the appearance parameters may be different in appearance from the physical object in the video data (for example, there may be some differences when the appearances match, and the difference is greater when the appearances are different). Therefore, in order to better render the virtual props and improve the sense of reality, in a possible implementation, the method further includes:

[0077] S41: Determine the appearance difference between the entity object in the video data and the target game object according to the game image parameters and the appearance parameters of the target game object.

[0078] S42: adjusting the game image parameters of the target game object according to the appearance difference to obtain adjusted image parameters adapted to the appearance parameters.

[0079] The corresponding S204 includes: rendering the virtual props corresponding to the target game object on the physical object in the video data according to the adjusted image parameters to obtain the target video.

[0080] The appearance difference identifies the difference in appearance of the target game object relative to the physical object in the video data. The game image parameters of the target game object are adjusted according to the determined appearance difference to obtain the bar image parameters. The adjusted image parameters can identify the mapping relationship between the virtual props and the physical object in the video data, so that when the virtual props of the target game object are rendered on the physical object in the video data, the virtual props that originally have a difference in appearance can be rendered fit on the physical object in the video data and in a coordinated proportion with the physical object, so that the picture is more beautiful and has a better sense of reality.

[0081] It can be seen that the game environment image of the target game can be displayed by setting the display area. When the physical object enters the display area, video data can be collected for the display area. Since the game environment image displayed by the display area serves as the background of the physical object, the physical object in the video data is like being in the scene of the target game, which plays a role of telepresence. By identifying the appearance parameters of the physical object in the video data, the target game object can be determined from multiple game objects of the target game based on the appearance parameters, and the virtual props corresponding to the target game object can be rendered on the physical object in the video data to obtain the target video. Since the target game object is determined based on the appearance parameters, it has a correlation with the physical object in appearance, which plays the role of the physical object playing the target game object in the target game scene, which not only brings a better immersive experience, but also has fun.

[0082] In a possible implementation, S203 includes:

[0083] S2031: Obtain game image parameters corresponding to the multiple game objects respectively.

[0084] S2032: Perform a similarity comparison between the appearance parameters and the game image parameters, and determine the target game object from the game objects whose similarity meets a threshold.

[0085] The game image parameters can identify the image characteristics of the corresponding game object, and the appearance parameters can reflect the appearance characteristics of the physical object. Therefore, the similarity comparison can be performed based on the appearance parameters and the game image parameters to determine the target game object from the game objects whose similarity meets the threshold. The threshold can be set to a relatively high value.

[0086] There can be at least one game object whose similarity meets the threshold. If there are multiple game objects, the one with the highest similarity can be randomly selected or selected as the target game object.

[0087] In order to further improve the similarity between the entity object and the target game object, when the entity object is a human body, the gender parameter of the human body can be introduced as a matching basis. The gender parameter can identify male or female.

[0088] In a possible implementation manner, the method further includes: acquiring a gender parameter of the human body.

[0089] Accordingly, S203 includes:

[0090] S2033: Determine a pending game object that meets the gender parameter from the multiple game objects.

[0091] S2034: Determine, based on the appearance parameters, a target game object that matches the entity object from among the pending game objects.

[0092] This scenario mainly involves target games that distinguish between game objects by gender. Based on the gender identified by the gender parameters of the entity object, a pending game object with the same gender can be determined from multiple game objects, and then the target game object can be determined from the pending game objects through the appearance parameters of the entity object. The determined target game object is consistent with the gender of the entity object, thereby improving the user experience.

[0093] In some application scenarios, the number of entity objects included in the video data is not limited, and one or more entity objects may be included in the display area. Different processing methods may be used for different numbers of entity objects.

[0094] In a possible implementation, the method further includes:

[0095] S11: Determine the object quantity of the physical objects in the video data, and if the object quantity is the specified quantity, execute S12; if the object quantity is not the specified quantity, execute S14.

[0096] The present application does not limit the specific value of the specified number, for example, it can be 1, or an integer greater than 1. The number of objects can be the number of entity objects of a certain type (for example, including several human bodies), or the total number of entity objects of all types (for example, the total number of human bodies and animals).

[0097] S12: Identify the actions performed by the entity object in the video data.

[0098] S13: In response to the action being a designated action, the operation of rendering the virtual props corresponding to the target game object on the physical object in the video data to obtain the target video is executed.

[0099] S14: Execute S204 to render the virtual props corresponding to the target game object on the physical object in the video data to obtain the target video.

[0100] When the number of entity objects reaches the specified number, it is necessary to determine whether to perform the operation of obtaining the target video based on whether the action performed by the entity object meets the requirements. This mechanism can bring additional interactive experience to the offline experience, and improve the sense of participation of the entity object by guiding the entity object to perform the specified action.

[0101] This application does not limit the specified action, which can be related to the type of physical object, the specified number, or the target game. For example, when the physical object is a human body, if the specified number is 1, the specified action can be a single-person action such as opening your hands and making a heart shape. If the specified number is an integer greater than 1, the specified action can be a combination action or multiple single-person actions. Among them, the combination action can be an action that requires a specified number of physical objects to work together to complete a complete action, such as multiple people holding hands, multiple people making a heart shape, etc.

[0102] For the human body as an entity object, the actions made by the entity object in the video data can be identified by identifying the positions of the key points of the human body's joints.

[0103] In a possible implementation, S12 includes:

[0104] S121: Determine the joint key point positions of the specified number of human bodies in the video data.

[0105] S122: Identify the actions performed by the specified number of human bodies in the video data according to the joint key point positions.

[0106] The joint key points of the human body are used to identify the joints of the human body, such as the shoulder, neck, elbow, ankle, pelvis and other joints that can produce displacement and rotation, and the joint key point positions are used to identify the positions of the corresponding joint key points. Based on the joint key point positions, the movements and postures of the human body can be recognized.

[0107] By identifying the video frames included in the video data, the positions of the key joint points of the human body in each video frame can be determined, and then the actions made by a specified number of human bodies in the video data can be determined through the timing relationship between the key joint point positions and the video frames.

[0108] The position of the corresponding joint key point can be selected for identification based on the specified action to be identified, or it can be identified based on the default position of the joint key point, and this application does not limit this.

[0109] In the embodiments of the present application, in addition to determining the corresponding target game object based on the appearance characteristics of the physical object, a richer interactive method can also be provided, such as rendering virtual props for the physical object based on the game object selected by the user. In other words, in addition to providing AR rendering of game objects that match the appearance of the physical object, AR rendering based on the specified object can also be performed based on the user's selection.

[0110] Therefore, in a possible implementation manner, the method further includes:

[0111] S21: Obtain a game object selection instruction for a specified object, where the specified object is one of the multiple game objects.

[0112] S22: Rendering the virtual prop corresponding to the designated object on the physical object in the video data to obtain a designated video.

[0113] Through the above method, users can select any game object they want to play through a physical object, which improves the diversity and selectivity of virtual prop rendering. For example, when a user as a physical object wants to play a game object that does not match his or her appearance, he or she can generate a game object selection instruction by selecting the game object, and then the terminal device renders the virtual props corresponding to the specified object on the physical object in the video data to obtain a specified video. This meets the needs of users who want to play game objects of different genders and appearances.

[0114] Next, a specific scenario in which the specified number is 1 and the specified action is "open both hands" is used as an example for explanation.

[0115] The terminal device can recognize the designated action of "opening both hands" through the recognition model, and the recognition model can recognize the positions of multiple key joints of the human body. Figure 3 In the scene shown, for the right hand of the human body, by identifying the positions of the key joint points 6 (right shoulder), 8 (right elbow) and 10 (right wrist), it can be determined whether the human body has made the action of opening the right hand. Similarly, by identifying the positions of the key joint points 5 (left shoulder), 7 (left elbow) and 9 (left wrist), it can be determined whether the human body has made the action of opening the left hand. Thus, when the conditions are met, it can be identified that the human body has made the action of "opening both hands".

[0116] For example, by calculating the angles of the wrist and elbow and the angles of the shoulder and elbow, it is determined as "open hands" when the angles of the left and right shoulders and elbows are less than 180 degrees and the angles of the wrist and elbow are greater than 25 degrees. The code can be as follows:

[0117] if(left_shoulder_angleA>25&&left_shoulder_angle A<180

[0118] &&left_shoulder_angleB>25&&left_shoulder_angleB<180

[0119] &&right_shoulder_angleA>25&&right_shoulder_angleA<180

[0120] &&right_shoulder_angleB>25&&right_shoulder_angleB<180){

[0121] armOpen=1;

[0122] }else{

[0123] armOpen = 0;

[0124] }

[0125] A and B refer to left and right respectively.

[0126] After determining that the human body has made the action of "opening both arms", the position and size of virtual props and virtual special effects can be calculated by identifying the positions of the key joint points 6 (right shoulder), 12 (right hip), 16 (right ankle), and 5 (left shoulder), so as to match different styles of props and special effects for human bodies in different positions and heights.

[0127] The terminal device of the present application can have different props and special effects matched to human bodies of different positions and heights.

[0128] Therefore, in a possible implementation manner, the method further includes: determining a video position of the entity object in the video data.

[0129] Correspondingly, S204 includes: determining a display form corresponding to the virtual prop according to the video position; and rendering the virtual prop on the physical object in the video data based on the display form to obtain a target video.

[0130] The physical object can be located at any position in the display area, and the position is recorded as the video position in the video data. Accordingly, in the collected video data, the size of the same physical object will be different when it is located at different video positions in the video data.

[0131] Therefore, in order to better render the virtual props on the physical object in the video data, at least two different display forms can be set for the virtual props corresponding to different game objects based on the distance and offset relationship between the video position of the physical object and the video acquisition device. Before rendering, based on the video position of the physical object in the video data, the display form corresponding to the virtual prop is selected, and the virtual prop is rendered on the physical object in the video data according to the display form to obtain the target video, thereby improving the rendering realism of the virtual prop.

[0132] like Figure 4 As shown, the left and right images correspond to different human bodies (physical objects), such as the human body in the left image is taller than the human body in the right image, and the human body is standing further back than the human body in the right image. The special effect in the left image is larger than that in the right image, and the virtual props of the corresponding target game object are rendered on the human body through the flying dragon flying towards the human chest in the special effect.

[0133] The code could look like this:

[0134] let left=(left_hip.y-left_shoulder.y) / 2+(left_shoulder.y);

[0135] let humanMidx=1920-(1920-right_shoulder.x)+(1920-right_shoulder.x)-(1920-left_shoulder.x)) / 2;

[0136] let humanMidy;

[0137] if(left_shouldlr.y>right_shoulder.y){

[0138] humanMidy=left_shoulder.y+(left_shoulder.y-right_shoulder.y) / 2;

[0139] }else{

[0140] humanMidy=right_shoulder.y+(right_shoulder.y-left_shoulder.y) / 2;

[0141] }

[0142] let topHeight=1080-humanMidy;

[0143] Among them, x and y represent the horizontal and vertical coordinates of the joint key point position respectively.

[0144] The above code calculates the midpoint of the left and right shoulders and hips of the human body as the midpoint of the virtual props, and calculates and sets the proportion of the virtual props according to the distance from the shoulders to the feet.

[0145] The determined target video may be an augmented reality (AR) video without music. AR technology is a technology that cleverly integrates virtual information with the real world. It simulates computer-generated text, images, three-dimensional models, music, videos and other virtual information and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0146] The interface for making videos can be called to generate the target video. The interface can be used to record the dynamic content of the canvas in the video data. The canvas draws the real content captured by the camera and virtual props (or virtual special effects, etc.). The content for the human body and the three-sided screen is the real content captured by the camera, and the virtual props are multiple transparent AR videos drawn at specified positions on the canvas. Finally, an AR video in WEBM format (i.e., the target video) will be generated, which reflects the interaction between the human body and the virtual props. Specifically, the recorded information can be stored in the data array, and the data can be converted into a file after the recording is completed to obtain the target video.

[0147] Since the aforementioned target video may be silent in some scenarios, audio and video fusion is required to integrate background music into the target video. The ffmpeg tool used for audio and video fusion (an open source free software that can perform recording, conversion, and streaming functions of audio and video in multiple formats) can be used to achieve the fusion of silent WEBM+MP3 (background audio) in this solution. The acquired silent video and the server audio are fused using the ffmpeg command to obtain the target video with background sound, which can be a file in MP4 format.

[0148] After the target video is generated, the user can also be provided with a way to obtain the target video, so that the user can save the target video on his or her terminal and watch or share it again when needed.

[0149] In a possible implementation, the method further includes:

[0150] S31: Save the target video in a storage location.

[0151] S32: Generate a corresponding address code based on the storage location, and display the address code.

[0152] S33: If a video download request sent by the terminal device is obtained based on the address code, the target video is returned to the terminal device from the storage location.

[0153] After the target video is generated, it can be saved. The storage location for saving the target video can be locally on the terminal device or on a server in the cloud.

[0154] After the target video is stored, an address code may be generated based on the storage location, and the address code may be in a digital form such as a QR code or a bar code.

[0155] The user can generate and send a video download request by scanning the address code through his terminal device such as a mobile phone. The terminal device as the aforementioned computer device can obtain the video download request and return the target video to the user's terminal device from the storage location.

[0156] For example, after the user scans the QR code as the address code through the mini program, the terminal device obtains the relevant parameters and uses them to make a network request. Then it determines whether the storage address of the target video (which can be in the form of a link) is valid. If the link is valid, the target video is accessed and the user is asked whether to save it. If the link is invalid, the request continues until the target video in WEBM format on the cloud is successfully converted to MP4 format and can be accessed.

[0157] based on Figure 5a and Figure 5bContinue to explain the video processing method provided by this application, such as Figure 5b As shown, the offline scene includes a terminal device 100, a camera 200, a three-sided screen 300 and an operation console 400. Among them, the specified number is 1, and the specified action is to open both hands.

[0158] In other words, there are two hosts and three screens in total. The two hosts are the console and the terminal device. The console is also used as a server. The console is used to select the interactive experience mode and confirm or delete the final generated video. The terminal device is used to record the immersive interactive AR video of the user and the hero special effects.

[0159] The three screens are respectively connected to the operation end and the display end, as well as a large external screen that is synchronized with the display end in real time. In addition, in order to better restore the game scene, three screens are also configured in the offline environment to play the game scene in a loop. Users can immerse themselves in the pleasure of becoming a hero on the three screens.

[0160] The user can be used as an entity object, and the camera 200 can capture video and obtain video data when the user is in the display area 300 of the three-screen logo. The terminal device 100 recognizes and matches the video data and renders the virtual props of the target game object on the user in the video data.

[0161] Before implementing the embodiment of the present application, the display interface 10 of the operation console 400 can be as follows: Figure 6a As shown, it includes steps one, two, and three for participating in the offline experience, as well as single-player and multiplayer options.

[0162] Among them, step 1 is used to instruct the user to enter the video acquisition area (i.e., the display area), step 2 is used to instruct the virtual prop rendering, and step 3 is used to instruct the scanning code to obtain the target video. Users can also choose single or multiple people based on the number of participants.

[0163] like Figure 5a As shown, in the case of a single person participating, by triggering Figure 6a Click the "single player" control to start, or by triggering Figure 6a Click the Multiplayer controls to start.

[0164] After clicking Start, the console 400 displays the following Figure 6b The content of the video is displayed in the display area 300, and the user is instructed to enter the display area 300 by "Please go to the photo area for interaction". The camera 200 collects the corresponding video data, wherein the content displayed in the display area 300 can be as follows Figure 7 shown.

[0165] The operating table 400 can also display Figure 6cThe content instructs users to perform specified actions through patterns and text such as "Please raise your arms and summon the dragon."

[0166] The terminal device 100 determines whether the user opens his hands based on the video data. If the recognition is successful, it determines whether to click on the special effect, that is, whether the virtual props need to be rendered.

[0167] When determining the click special effect, the terminal device 100 renders the corresponding virtual props on the user in the video data based on the determined target game object to generate a target video. The target video can be Figure 8 As shown, Figure 8 What is shown is a situation where two users are in the display area 300 at the same time, which is displayed through the display interface 20 of the terminal device 100 .

[0168] Figure 8 The virtual props of the target game object matched by the user on the left are clothing and headgear, and the virtual props of the target game object matched by the user on the right are machine guns hanging on the shoulders. Figure 6a The controls that trigger "multiplayer" in .

[0169] After the terminal device 100 processes the target video, it stores it in a storage location and generates an address code based on the storage location. The address code can be Figure 5b The display is performed on the terminal device 100 in the illustrated scene, and may also be performed on the operation console 400 .

[0170] The user scans the address code through the mini program. If authorized, the terminal device 100 returns the target video to the user and saves it in the photo album of the user's mobile phone. If not authorized, the target video cannot be obtained.

[0171] The process of single-player experience can be found in Figure 9a , the user clicks the single-player shooting button on the console. The user then stands in the best shooting area in the display area and opens his hands. Based on the transmitted video data, the current environment captured by the camera is displayed on the large screen of the terminal device. After recognizing the action of opening his hands, the terminal device will adapt the virtual props special effects of the game hero according to the user's position and height, and render them together with the content captured by the camera. The user can then interact with the hero special effects. After the interaction is over, the target video of the entire AR interactive experience will be generated. After the target video is shot, the audio can be fused. The target video is displayed on the console. After the user clicks to save the video, the terminal device saves the target video and generates the corresponding address code. The address code is transmitted to the display end for display, and the user can scan the code with the applet to save it.

[0172] The process of multi-person experience can be found in Figure 9b, the user clicks the multi-player shooting button on the console. In some cases, when multiple people open their hands and use tendorflow.js to identify them, they will directly interfere with each other and affect the recognition of actions. Therefore, it is optional to use the method of actively triggering the camera and hero special effects on the console. The interaction after turning it on is consistent with the single-player experience, so I will not go into details one by one.

[0173] The embodiments of the present application can be implemented based on the web, and an immersive game role experience can be created through the combination of technology and offline environment.

[0174] In the aforementioned Figure 1-9b On the basis of Fig.10 The structure diagram of a video processing device provided in an embodiment of the present application is as follows. The video processing device 1000 includes an acquisition unit 1001, a determination unit 1002, and a rendering unit 1003:

[0175] The acquisition unit 1001 is used to acquire video data collected for a display area, where the display area is used to display a game environment image of a target game, and where there is a physical object in the display area;

[0176] The determining unit 1002 is used to determine the appearance parameters of the entity object in the video data;

[0177] The determining unit 1002 is further configured to determine a target game object from among a plurality of game objects of the target game according to the appearance parameter;

[0178] The rendering unit 1003 is used to render the virtual props corresponding to the target game object on the physical object in the video data to obtain the target video.

[0179] In a possible implementation, the determination unit is further configured to determine, according to the appearance parameters, a target game object that matches the entity object among a plurality of game objects of the target game.

[0180] In a possible implementation manner, the determining unit is further configured to:

[0181] Determining the appearance difference between the entity object in the video data and the target game object according to the game image parameters and the appearance parameters of the target game object;

[0182] Adjust the game image parameters of the target game object according to the appearance difference to obtain the adjusted image parameters adapted to the appearance parameters;

[0183] The rendering unit is further used to render the virtual props corresponding to the target game object on the physical object in the video data according to the adjusted image parameters to obtain the target video.

[0184] In a possible implementation, if the entity object is a human body, the determination unit is further configured to determine at least one of facial features, height parameters, and body shape parameters of the human body in the video data as the appearance parameter.

[0185] In a possible implementation manner, the determining unit is further configured to:

[0186] Obtaining game image parameters corresponding to the multiple game objects respectively;

[0187] A similarity comparison is performed based on the appearance parameters and the game image parameters, and the target game object is determined from game objects whose similarity meets a threshold.

[0188] In a possible implementation, the acquisition unit is further used to acquire a gender parameter of the human body;

[0189] The determining unit is further configured to:

[0190] Determine a pending game object that meets the gender parameter from the multiple game objects;

[0191] According to the appearance parameters, a target game object matching the entity object is determined among the pending game objects.

[0192] In a possible implementation manner, the device further includes an identification unit:

[0193] The determining unit is further configured to determine the number of the entity objects in the video data;

[0194] The recognition unit is used to recognize the action performed by the physical object in the video data if the number of objects is a specified number;

[0195] The recognition unit is further configured to trigger the rendering unit in response to the action being a specified action;

[0196] The determining unit is further configured to trigger the rendering unit if the number of objects is not the specified number.

[0197] In a possible implementation manner, the physical object is a human body, and the recognition unit is further configured to:

[0198] Determine the joint key point positions of the specified number of human bodies in the video data;

[0199] The actions performed by the specified number of human bodies in the video data are identified according to the joint key point positions.

[0200] In a possible implementation manner, the acquiring unit is further configured to:

[0201] Obtaining a game object selection instruction for a specified object, wherein the specified object is one of the multiple game objects;

[0202] The virtual props corresponding to the designated object are rendered on the physical object in the video data to obtain a designated video.

[0203] In a possible implementation manner, the determining unit is further used to determine a video position of the entity object in the video data;

[0204] The rendering unit is also used for:

[0205] Determining a display form corresponding to the virtual prop according to the video position;

[0206] The virtual prop is rendered on the physical object in the video data based on the display form to obtain a target video.

[0207] In a possible implementation manner, the device further includes a storage unit, and the storage unit is used to:

[0208] Saving the target video in a storage location;

[0209] generating a corresponding address code based on the storage location, and displaying the address code;

[0210] If a video download request sent by a terminal device is obtained based on the address code, the target video is returned to the terminal device from the storage location.

[0211] In a possible implementation manner, the display area is a three-sided screen, and the three display screens of the three-sided screen are arranged perpendicular to each other.

[0212] It can be seen that the game environment image of the target game can be displayed by the set display area. When the physical object enters the display area, video data can be collected for the display area. Since the game environment image displayed by the display area is used as the background of the physical object, the physical object in the video data is like being in the scene of the target game, which plays a role of telepresence. By identifying the shape parameters of the physical object in the video data, the target game object that matches the physical object can be determined from multiple game objects of the target game based on the shape parameters, and the virtual props corresponding to the target game object are rendered on the physical object in the video data to obtain the target video. Since the target game object is similar to the physical object in appearance, the physical object rendered with virtual props in the target video will be more similar to the target game object, which plays the role of the physical object playing the target game object in the target game scene, not only bringing a better immersive experience, but also having fun.

[0213] The embodiment of the present application further provides a computer device, which is the computer device described above and may include a terminal device or a server, and the video processing device described above may be configured in the computer device. The computer device is described below in conjunction with the accompanying drawings.

[0214] If the computer device is a terminal device, see Fig.11 As shown, the embodiment of the present application provides a terminal device, taking the terminal device as a mobile phone as an example:

[0215] Fig.11 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.11 The mobile phone includes: a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. Those skilled in the art can understand that Fig.11 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0216] Combine the following Fig.11 A detailed introduction to the various components of the mobile phone:

[0217] The RF circuit 1410 may be used for receiving and sending signals during information transmission or communication or during a call. In particular, after receiving downlink information from a base station, the information is sent to the processor 1480 for processing. In addition, the uplink data is sent to the base station.

[0218] The memory 1420 may be used to store software programs and modules, and the processor 1480 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1420 .

[0219] The input unit 1430 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432. The touch panel 1431, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on the touch panel 1431 or near the touch panel 1431), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1431 may include two parts: a touch detection device and a touch controller.

[0220] The display unit 1440 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

[0221] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

[0222] The audio circuit 1460, the speaker 1461, and the microphone 1462 can provide an audio interface between the user and the mobile phone. The audio circuit 1460 can transmit the received audio data to the speaker 1461 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1460 and converted into audio data, and then the audio data is output to the processor 1480 for processing, and then sent to another mobile phone through the RF circuit 1410, or the audio data is output to the memory 1420 for further processing.

[0223] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1470. It provides users with wireless broadband Internet access. Fig.11 A WiFi module 1470 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0224] Processor 1480 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in memory 1420 and calling data stored in memory 1420.

[0225] In this embodiment, the processor 1480 included in the terminal device also has the following functions:

[0226] Acquire video data collected for a display area, where the display area is used to display a game environment image of a target game, and where there is a physical object in the display area;

[0227] Determining shape parameters of the physical object in the video data;

[0228] Determining, according to the appearance parameters, a target game object that matches the entity object among a plurality of game objects of the target game;

[0229] The virtual props corresponding to the target game object are rendered on the physical object in the video data to obtain the target video.

[0230] If the computer device is a server, the present application embodiment also provides a server, see Fig.12 As shown, Fig.12 The structural diagram of the server 1500 provided in the embodiment of the present application, the server 1500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1522 (for example, one or more processors) and a memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be short-term storage or permanent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1522 can be configured to communicate with the storage medium 1530 and execute a series of instruction operations in the storage medium 1530 on the server 1500.

[0231] The server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server 2003 or Windows Server 2008. TM , Mac OS X TM , UnixTM ,Linux TM , FreeBSD TM etc.

[0232] The steps performed by the server in the above embodiment can be based on Fig.12 The server structure shown.

[0233] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is used to execute the method provided in the above embodiment.

[0234] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method provided in the above embodiments.

[0235] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0236] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0237] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Moreover, based on the implementation methods provided in the above aspects, the present application can also be further combined to provide more implementation methods. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire video data collected for a display area, the display area is used to display a game environment image of a target game, and there is a physical object in the display area; the physical object is an actual object with entity; Determine the appearance parameters of the entity object in the video data; Obtaining game image parameters corresponding to a plurality of game objects of the target game respectively; A similarity comparison is performed based on the appearance parameters and the game image parameters, and a target game object matching the entity object is determined from the game objects whose similarity meets a threshold, so that the target game object and the entity object are related in appearance; the multiple game objects are game characters of different forms or types in the target game; the target game object is the game character played by the entity object; Based on the game image parameters and the appearance parameters of the target game object, the appearance difference between the entity object and the target game object in the video data is determined; Adjust the game image parameters of the target game object according to the appearance difference, and obtain the adjusted image parameters that are adapted to the appearance parameters; According to the adjusted image parameters, the virtual props corresponding to the target game object are rendered on the physical object in the video data to obtain the target video; the adjusted image parameters are used to identify the mapping relationship between the virtual props and the physical object in the video data, so that when the virtual props of the target game object are rendered on the physical object in the video data, the virtual props that originally have different appearances are rendered fit on the physical object in the video data; the virtual props corresponding to the target game object are used to identify the appearance characteristics of the target game object and distinguish it from other game objects.

2. The method according to claim 1, characterized in that: If the physical object is a human body, the determination of the physical object's appearance parameters in the video data includes: At least one of the face characteristics, height parameters, and body shape parameters of the human body in the video data is determined as the appearance parameter.

3. The method according to claim 1, characterized in that If the physical object is a human body, the method further includes: Obtaining gender parameters of the human body; According to the appearance parameters, a target game object matching the entity object is determined among the multiple game objects of the target game, including: Determine a pending game object that meets the gender parameters from the plurality of game objects; According to the appearance parameters, a target game object matching the entity object is determined in the pending game object.

4. The method according to claim 1, characterized in that The method further comprises: Determine the number of objects of the entity objects in the video data; If the number of objects is a specified number, the actions performed by the entity object in the video data are identified; In response to the action as a designated action, the operation of rendering the virtual props corresponding to the target game object on the entity object in the video data is performed to obtain the target video; If the number of objects is not the specified number, the operation of rendering the virtual props corresponding to the target game object on the entity object in the video data is performed to obtain the target video.

5. The method according to claim 4, characterized in that The physical object is a human body, and the recognition of the physical object's actions in the video data includes: Determine the location of joint key points of the specified number of human bodies in the video data; The specified number of actions made by the human body in the video data are identified based on the joint key point position.

6. The method according to claim 1, characterized in that The method further comprises: Obtain a game object selection instruction for a specified object, the specified object being one of the multiple game objects; The virtual props corresponding to the specified object are rendered on the entity object in the video data to obtain the specified video.

7. The method according to claim 1, characterized in that The method further comprises: Determine the video location of the entity object in the video data; The virtual props corresponding to the target game object are rendered on the entity object in the video data to obtain the target video, including: Determine the display form corresponding to the virtual props based on the video position; Based on the display form, the virtual prop is rendered on the physical object in the video data to obtain the target video.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Save the target video in a storage location; A corresponding address code is generated based on the storage location and the address code is displayed; If the video download request sent by the terminal device is obtained based on the address code, the target video is returned to the terminal device from the storage location.

9. The method according to any one of claims 1-7, characterized in that The display area is a three-sided screen, and the three display screens of the three-sided screen are arranged vertically to each other.

10. A video processing device, characterized in that: The device includes a acquisition unit, a determination unit and a rendering unit: The acquisition unit is used to acquire video data collected for a display area, the display area is used to display a game environment image of a target game, and there is a physical object in the display area; the physical object is an actual object with entity; The determination unit is used to determine the appearance parameters of the entity object in the video data; The determination unit is further used to obtain game image parameters corresponding to a plurality of game objects of the target game respectively; perform similarity comparison between the appearance parameters and the game image parameters, and determine a target game object matching the entity object from game objects whose similarity meets a threshold, so that the target game object and the entity object are related in appearance; the plurality of game objects are game characters of different forms or types in the target game; the target game object is a game character played by the entity object; The determining unit is further configured to determine the appearance difference between the entity object and the target game object in the video data according to the game image parameters and the appearance parameters of the target game object; and adjust the game image parameters of the target game object according to the appearance difference to obtain adjusted image parameters adapted to the appearance parameters; The rendering unit is used to render the virtual props corresponding to the target game object on the physical object in the video data according to the adjusted image parameters to obtain a target video; the adjusted image parameters are used to identify the mapping relationship between the virtual props and the physical object in the video data, so that when the virtual props of the target game object are rendered on the physical object in the video data, the virtual props that originally have different appearances are rendered fit on the physical object in the video data; the virtual props corresponding to the target game object are used to identify the appearance characteristics of the target game object and distinguish it from other game objects.

11. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method of any one of claims 1-9 according to instructions in the program code.

12. A computer-readable storage medium, characterized in that: The computer readable storage medium is used to store a computer program for performing the method of any one of claims 1-9.

13. A computer program product including instructions, characterized in that When it is run on a computer, the computer is caused to perform the method of any one of claims 1-9.

Images