Image processing method, electronic device, storage medium and computer program product

Abstract

The invention discloses an image processing method, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining a first video stream which is obtained by shooting the same site from a plurality of visual angles; if a target event is detected in the first video stream, generating a three-dimensional model of a target object associated with the target event; sending the first information to the client; wherein the first information at least comprises the first video stream and a three-dimensional model of the target object, and the three-dimensional model of the target object is used for being displayed on a playing interface of the first video stream so that a user can carry out corresponding operation. According to the scheme, the interestingness of watching the video can be improved, so that the requirement of watching the multi-view video in an immersive manner by the user is met.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] With the continuous development of multimedia technology and video capture equipment, multi-view video is increasingly widely used in sports events, virtual reality, and other fields. Related technologies support multi-view video viewing solutions that allow users to switch viewing angles by swiping left and right, zoom in or out by pinching with two fingers, and support intelligent tracking and zooming in / out based on specific people (such as star players) or objects (such as basketballs or footballs). However, existing solutions lack the fun of watching multi-view videos and fail to meet users' needs for immersive multi-view video viewing. Summary of the Invention

[0003] To address the related technical problems, embodiments of this application provide an image processing method, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides an image processing method, including:

[0006] Acquire a first video stream, which is obtained from multiple perspectives of the same location;

[0007] If a target event is detected in the first video stream, a 3D model of the target object associated with the target event is generated;

[0008] Send first information to the client; wherein the first information includes at least the first video stream and a three-dimensional model of the target object, the three-dimensional model of the target object being displayed on the playback interface of the first video stream for the user to perform corresponding operations.

[0009] This application also provides an electronic device, characterized in that it includes a processor and a memory for storing a computer program capable of running on the processor.

[0010] When the processor runs the computer program, it executes the steps of any of the above methods.

[0011] This application also provides a storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of any of the above methods.

[0012] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0013] In the image processing methods, electronic devices, storage media, and computer program products provided in the various embodiments of this application, by acquiring a first video stream and detecting a target event in the first video stream, a three-dimensional model of the target object associated with the target event can be generated, and first information can be sent to the client. The above scheme can identify the target event and generate a three-dimensional model of the target object associated with the target event. This allows the terminal running the client to display the three-dimensional model of the target object on the playback interface of the first video stream while playing image frames related to the target event in the first video stream. This enables users to understand the process of the target event while watching the video content of the first video stream and to perform corresponding operations on the three-dimensional model of the target object associated with the target event, thereby enhancing the enjoyment of watching the video and meeting the user's need for immersive multi-view video viewing. Attached Figure Description

[0014] Figure 1 This is a schematic flowchart of a data processing method according to an embodiment of this application;

[0015] Figure 2 Example diagram for obtaining the first video stream in an embodiment of this application;

[0016] Figure 3 This is a schematic diagram of another data processing method according to an embodiment of this application;

[0017] Figure 4 This is an example diagram illustrating the target object associated with the target event detected in an embodiment of this application.

[0018] Figure 5 This is an example diagram illustrating how a target object is associated with a target event through a worker-directed broadcast method, as described in this application embodiment.

[0019] Figure 6 This is an example diagram illustrating a picture-in-picture display of a video frame from a first video stream and a 3D model of a target object, as described in this application embodiment.

[0020] Figure 7 This is an example diagram illustrating how the video footage of the first video stream and the 3D model of the target object are displayed in a split-screen manner according to an embodiment of this application.

[0021] Figure 8 This is a schematic diagram of the structure of an image processing device according to an embodiment of this application;

[0022] Figure 9 This is a schematic diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation

[0023] In various embodiments of this application, by acquiring a first video stream and detecting a target event in the first video stream, a 3D model of the target object associated with the target event can be generated, and first information can be sent to the client. This scheme can identify the target event and generate a 3D model of the target object associated with the target event. This allows the terminal running the client to display the 3D model of the target object on the playback interface of the first video stream while playing image frames related to the target event in the first video stream. This enables users to understand the process of the target event while watching the video content of the first video stream and to perform corresponding operations on the 3D model of the target object associated with the target event, thereby enhancing the viewing experience and meeting the user's need for immersive multi-view video viewing.

[0024] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0025] This application provides an image processing method applied to an electronic device, which includes, but is not limited to, one or more of the following: a server, a server-side application for multi-view video playback, or a device with a server-side application deployed for video viewing. Figure 1 As shown, the method includes:

[0026] Step 101: Obtain the first video stream, which is obtained from multiple perspectives of the same location.

[0027] Here, video streams synchronously captured from multiple perspectives by multiple cameras positioned around the same venue can be acquired or received. These video streams are then processed (e.g., stitching and transcoding) to obtain the first video stream. The term "camera" can be replaced with "camera." The venue refers to the physical space where a competition or event takes place, including competition venues and / or event venues. Competition venues are specifically designed for hosting competitive activities, such as football fields, basketball courts, sports fields, athletic fields, swimming pools, and boxing rings. These venues have clearly defined boundaries and standard dimensions, facilitating positioning and modeling. "Perspective" can be understood as the shooting angle, and the first video stream can be understood as a multi-view video stream or multi-view video. Compared to a monocular video stream, a multi-view video stream provides richer spatial information, aiding in subsequent target event identification and 3D model construction.

[0028] For example, such as Figure 2 As shown, in a game, multiple cameras can be set up around the perimeter of the court, capturing 360-degree footage from various locations such as the stands, behind the basket, and the player entrance tunnel, thus creating a multi-view video stream covering the entire game—the first video stream. This multi-view video stream can not only be used for real-time playback but also as input for subsequent 3D modeling and 3D pose estimation.

[0029] It should be noted that electronic devices can obtain complete visual coverage of all objects in the venue through the first video stream, thereby improving the accuracy of subsequent target event detection and the quality of 3D model generation.

[0030] Step 102: If a target event is detected in the first video stream, a 3D model of the target object associated with the target event is generated.

[0031] Here, the electronic device detects whether a target event exists in the first video stream. The target event can be identified by an intelligent directing system or specified or marked by a manual directing method. If a target event is detected in the first video stream, image frames related to the target event are acquired in the first video stream, and the target object associated with the target event is identified in the image frames related to the target event. Based on the image of the target object in the image frames related to the target event, a 3D model of the target object associated with the target event is generated. The image frames related to the target event can include at least the image frame in which the target event occurs, and can also include a certain number of image frames before or after the target event occurs; for example, the image frames related to the target event can include multiple image frames of a specific duration (e.g., 15 seconds) in which the target event occurs. Image frames can be replaced with video frames, and a video frame can include multiple image frames.

[0032] A target event and a target object have a corresponding or associated relationship. There can be one or more target events, and the same target event can be associated with one or more target objects. A target event refers to an event in the first video stream that has specific significance and can be intelligently identified, such as a goal, shot, foul, score, or completion of a competitive event. Target events can be described as key events, exciting events, or memorable moments. A target object refers to a specific entity associated with a target event. This entity can be a participant (e.g., a player, athlete) and / or a tool used in the competition (e.g., a basketball, soccer ball, shot put, weightlifting equipment). In practical applications, target objects typically refer to the main people and / or objects involved in the target event, such as a goal scorer, a passer, a referee, or the ball. The target object is the core content that the electronic device focuses on; the behavior or state changes of the target object directly relate to the occurrence of the target event.

[0033] It should be noted that when an electronic device detects a target event, it triggers the process of generating a 3D model of the target object associated with that event. For example, in a football match, when a player takes a shot and successfully scores, the electronic device will automatically identify the event of a shot and a successful goal in the football match, and use that player as a target object to generate a 3D model of that player; alternatively, it can use both the player and the ball as target objects to generate separate 3D models of the player and the ball, so that the process of the player taking the shot can be displayed later.

[0034] A 3D model is a digital, three-dimensional image constructed using 3D reconstruction algorithms based on the features of a target object as shown in the first video stream. 3D models can be used for various purposes, including visualization, animation simulation, and user interaction. For example, after a goal is scored, electronic devices can analyze the player's posture changes through the first video stream, extract the player's movement trajectory, and generate a dynamic 3D model based on these trajectories. This 3D model can then display the goal-scoring process in 360 degrees.

[0035] In practical applications, intelligent broadcasting systems can detect or identify target events in the first video stream, or identify target events specified by a human director within the first video stream. Intelligent broadcasting systems can make comprehensive judgments based on multi-dimensional information such as ball status tracking, player positional relationships, and action patterns.

[0036] It should be noted that in practical applications, the 3D model of the target object associated with the target event can be a 3D model constructed based on the target object's feature information, such as a 3D model constructed based on the target object's feature information contained in an image frame that detects the target event; or, in the case of constructing a 3D model of the target object based on its feature information, a dynamic 3D model obtained by driving or rendering the 3D model of the target object based at least on the target object's 3D pose parameters. Furthermore, the electronic device can also drive or render the 3D model of the target object based on the target object's 3D pose parameters and a first control command for the 3D model of the target object. The 3D pose parameters of the target object can be determined based on the 2D image of the target object contained in one or more image frames associated with the target event. The first control command for the 3D model of the target object can be pre-configured or based on one or more of the following: voice information input by the anchor, director, or narrator; touch commands; and control commands triggered by peripherals such as a mouse. The first control command can include 360-degree multi-angle viewing, rotating the viewing angle, slowing down, speeding up, moving, zooming in, zooming out, adjusting the background color, etc.

[0037] Step 103: Send first information to the client; wherein the first information includes at least the first video stream and a three-dimensional model of the target object, and the three-dimensional model of the target object is used to display on the playback interface of the first video stream for the user to perform corresponding operations.

[0038] Here, the electronic device can fuse the first video stream and the 3D model of the target object to obtain first information. For example, it can embed or insert image frames of the 3D model of the target object into image frames associated with the target object in the first video stream to obtain first information. Alternatively, it can establish a relationship between the 3D model of the target object and the image frames corresponding to the target in the first video stream, and generate first information based on the first video stream and the 3D model of the target object. The first video stream and the 3D model of the target object in the first information are decoupled. Upon obtaining the first information, the electronic device can send the first information to a client or to a terminal running the client via network protocols (such as RTMP, HLS, WebRTC, etc.). This allows the terminal to display the 3D model of the target object while playing the first video stream, enabling users to perform corresponding operations on the 3D model. Users can flexibly control the display effect of the model according to their needs, thereby enhancing the user's viewing experience and improving immersion and interactivity. The client supports playing video streams, and the terminal includes, but is not limited to, one or more of mobile phones, tablets, and smart TVs. Network protocols include, but are not limited to, one or more of the following: Real-Time Messaging Protocol (RTMP), HTTP Live Streaming (HLS), and Web Real-Time Communication (WebRTC). Corresponding operations refer to user commands issued on the 3D model of a target object within the playback interface of the first video stream, including but not limited to one or more of the following: rotating the viewpoint, slowing down, speeding up, zooming in, zooming out, and moving the position. For example, while watching the video content of the first video stream, a user can choose to place the 3D model of the target object on the left side of the screen and zoom in / out to observe the player's movement details more clearly; or they can use a swipe gesture to rotate the 3D model of the target object 360 degrees to view the player's movement trajectory from different angles.

[0039] For example, in a live football match, electronic devices can send the first video stream along with a 3D model of the target object to a client installed on the user's terminal. While watching the match through the terminal, the user can rotate, scale, and move the 3D model of the target object using gestures or touch commands.

[0040] In practical applications, while the terminal is playing the first video stream, it can display the 3D model of the target object alongside the first video stream in a picture-in-picture or split-screen format. For example, when a target event appears in the first video stream, the 3D model of the target object associated with that event is overlaid in a small window in a corner of the main screen. This allows the user to view the details of the target object's 3D model without interrupting the first video stream, enhancing the user's interactive experience. Furthermore, users can switch to different display modes, such as slow-motion playback or tactical breakdown, by clicking on the target object's 3D model or selecting menu items.

[0041] In practical applications, the terminal can utilize a front-end rendering engine (such as Unity, Unreal Engine, Three.js, etc.) to achieve real-time rendering and interactive control of the target object's 3D model. The terminal can record all user actions on the target object's 3D model and instantly reflect these actions back to the model, ensuring smooth and interactive display. For example, in a scenario where a user is watching a basketball game corresponding to the first video stream, they can pause the target object's 3D model during playback and use gestures to zoom in on the target object's hand movements for a closer look at shooting techniques.

[0042] In this embodiment, multi-view video streams can be acquired, and when a target event is detected in the multi-view video streams, a 3D model of the target object associated with the target event can be generated. The first video stream and the 3D model of the target object are then sent to the client, so that users can perform corresponding operations on the 3D model of the target object related to the target event while watching the multi-view video. This not only improves the visualization of the event content but also enhances the user's sense of participation and interactivity, increasing the fun of watching multi-view videos. In turn, it can improve the intelligent viewing experience of digital-real integration (the integration of digital technology and the real economy).

[0043] In one embodiment, generating a 3D model of a target object associated with the target event if a target event is detected in the first video stream includes:

[0044] If a target event is detected in the first video stream, feature information of the target object associated with the target event is extracted from the first video stream.

[0045] Based on the feature information of the target object, a three-dimensional model of the target object is performed to obtain the three-dimensional model of the target object.

[0046] Here, when a target event is detected in the first video stream, an image frame of the target object associated with the target event is determined in the first video stream; in the image frame of the target object associated with the target event, the feature information of the target object associated with the target event is extracted, and based on the feature information of each target object associated with the target event, a 3D model of each target object is performed to obtain a 3D model of each target object.

[0047] Among them, feature information refers to the key visual data extracted from the first video stream to describe the target object, such as contours and bounding boxes, which are used to determine the shape, size and contour of the target object.

[0048] 3D modeling refers to the process of constructing the three-dimensional geometric structure of a target object using feature information and giving the target object dynamic behavior.

[0049] For example, deep learning or other image processing techniques can be used to extract player contours, bounding boxes, and other feature information from the first video stream. Then, a parametric human model (such as SMPL, SCAPE, etc.) can be used to initialize the player's 3D model based on this feature information. Parametric human models include, but are not limited to, Skinned Multi-Person Linear Models (SMPL) and / or Shape Completion and Animation of People (SCAPE).

[0050] In this embodiment, by extracting the feature information of the target object associated with the target event from the first video stream, and obtaining a three-dimensional model based on the feature information of the target object, the motion trajectory and posture changes of the target object can be reproduced, achieving high-precision restoration and visualization of key actions. This allows viewers to watch key actions in the competition from any angle, enabling in-depth analysis and interactive display of multi-view video content, thereby enhancing the functionality and fun of the intelligent viewing system and providing a richer and more personalized multi-view video experience.

[0051] After constructing the 3D model of the target object, the 3D model can be rendered so that users can perform corresponding operations on the 3D model of the target object without interrupting the viewing of multi-view videos. Based on this, in one embodiment, the step of generating a 3D model of the target object associated with the target event if a target event is detected in the first video stream further includes:

[0052] Extract the three-dimensional pose parameters of the target object from the first video stream, and obtain the first control command for the three-dimensional model of the target object;

[0053] Based on the three-dimensional pose parameters of the target object and the first control command, the three-dimensional model of the target object is rendered to obtain the rendered three-dimensional model of the target object.

[0054] Here, image frames of the target object associated with the target event are identified in the first video stream. From these image frames, the 3D pose parameters of the target object are extracted. For example, 2D image features of the target object are extracted, and the 3D pose parameters can be determined based on these 2D image features using a defined algorithm. The defined algorithm includes, but is not limited to, one or more of the following: feature matching and triangulation reconstruction algorithms, such as Multi-View Pose (MVPose); algorithms based on spatial voxelization, such as Voxel-based Multi-Camera 3D Human Pose Estimation; and algorithms based on graph convolutional networks. The 3D pose parameters of the target object can be understood as the coordinate information of key points of the target object in 3D space. These parameters reflect the target object's posture, motion trajectory, and dynamic changes in the motion trajectory, such as arm swings, leg movements, and head rotations. 3D pose parameters provide a more accurate description of motion than 2D pose.

[0055] Obtaining a first control command for the 3D model of a target object may include: obtaining a first control command for the 3D model of the target object from a local configuration, where the first control command may be a pre-defined control command; and / or, obtaining a first control command for the 3D model of the target object triggered by a first user on the playback interface of the first video stream. For example, while watching the first video stream, the first user records commentary on the first video stream. During the commentary on the target event, the user performs corresponding operations on the 3D model of the target object displayed on the playback interface of the first video stream, triggering a first control command for the 3D model of the target object. The first control command may include one or more of the following: 360-degree multi-angle viewing, rotating the viewpoint, slowing down, speeding up, moving, zooming in, zooming out, and adjusting the display mode of the 3D model of the target object. The first user can be understood as a broadcaster; the first user can interact with the 3D model of the target object through clicks or gestures to trigger the first control command, thereby enhancing the viewer's understanding of the target event process and the viewing experience.

[0056] Once the electronic device obtains the 3D posture parameters of the target object and the first control command for the 3D model of the target object, it renders the 3D model of the target object based on the 3D posture parameters and the first control command for the 3D model of the target object, resulting in a rendered 3D model of the target object. Rendering can be replaced by driving. For example, a graphics engine (such as Unity, Unreal Engine, etc.) can be used to realize the dynamic rendering and interactive control of the 3D model of the target object, so as to present the 3D model of the target object in a visual way. The process of rendering the 3D model of the target object includes operations such as animation driving of the 3D model, adding lighting and shadow effects, and material mapping, to ensure that the 3D model of the target object presents a natural and realistic visual effect from different perspectives. Of course, animation production techniques can also be combined to smooth the movement of the 3D model to improve the user's visual experience. For example, at the moment of a goal, the electronic device can generate a replay of the target object's action from different angles based on the 3D posture parameters of the target object, and allow the first user to adjust the perspective, playback speed, and display mode of the 3D model of the target object through the first control command to highlight key tactical details, thereby helping the audience better understand the process and tactical arrangements of key events in the game. The display method can be to display the first video stream and the 3D video of the target object in a picture-in-picture or split-screen format.

[0057] It should be noted that, in this embodiment, the process of generating a 3D model of the target object may include multiple technical steps such as multi-view collaborative analysis, pose estimation, and model-driven processing. First, the electronic device can acquire image frames of the target object from multiple viewpoints in the first video stream, and perform collaborative processing on these image frames to obtain the complete outline of the target object. When the image of the target object is occluded from a certain viewpoint, it can be supplemented and inferred from 2D images of the target object in other viewpoints to solve the occlusion problem, thereby improving the accuracy and completeness of the 3D modeling process. Then, using deep learning or traditional image processing methods, key points of the target object are extracted from the image frames from multiple viewpoints, and the 3D pose parameters of the target object are determined based on these key points using a set algorithm. Finally, the 3D pose parameters of the target object are mapped onto the 3D model of the target object, realizing the dynamic driving or rendering of the 3D model. For example, in a basketball game scenario, the electronic device can detect a player performing a dunk in the first video stream, automatically generate a 3D model of the player performing the dunk, and restore the motion trajectory of the 3D model of the player performing the dunk as a visualized 3D animation for viewers to watch.

[0058] In this embodiment, based on the 3D pose parameters of the target object and the first control command for the 3D model of the target object, the 3D model of the target object is rendered, i.e., driven. This enables dynamic rendering of the 3D model, allowing the 3D model to display the realistic actions of the target object in the target event. Furthermore, the display method of the 3D model of the target object can be flexibly adjusted based on the first control command to meet the viewing needs of different scenarios. This method breaks the limitation of traditional videos that can only be played from a fixed angle, allowing viewers to observe the process of the target event from any angle, thus enhancing the viewing experience.

[0059] To reduce the error between the 3D model and the 2D image of the target object, in one embodiment, after generating the 3D model of the target object, the method further includes:

[0060] In the first video stream, extract two-dimensional image features of the target object from multiple perspectives;

[0061] Based on the two-dimensional image features of the target object from multiple perspectives, the three-dimensional model of the target object is adjusted.

[0062] Here, in the first video stream, two-dimensional images of the target object from multiple perspectives can be acquired, and two-dimensional image features of the target object from multiple perspectives can be extracted. Based on the two-dimensional image features of the target object from multiple perspectives, the three-dimensional model of the target object can be optimized and updated, so that the three-dimensional model of the target object is closer to the posture and shape in the real scene.

[0063] Among them, electronic devices can use optimization algorithms (such as gradient descent, least squares, etc.) to iteratively adjust the relevant parameters (such as shape and attitude parameters) of the three-dimensional model of the target object, thereby minimizing the error between the three-dimensional model of the target object and the two-dimensional image features, so that the three-dimensional model of the target object has high realism and consistency from different perspectives, and can also provide a basis for subsequent tactical analysis and interactive viewing.

[0064] In scenarios where a live commentator is providing commentary on a sporting event, electronic devices can fuse a first video stream, the commentary content, and a 3D model of the target object before sending them to the client. This allows viewers to interact with the 3D model of the target object within the first video stream's playback interface. Based on this, in one embodiment, sending the first information to the client includes:

[0065] Obtain second information, which includes the narration content of the first video stream or the narration content of the target event;

[0066] Based on the image frames associated with the second information in the first video stream and the image frames associated with the three-dimensional model of the target object in the first video stream, the second information, the three-dimensional model of the target object, and the first video stream are fused to obtain a second video stream. The three-dimensional model of the target object is used to display the second video stream in a set manner on the playback interface.

[0067] The second video stream is sent to the client.

[0068] Here, the electronic device acquires the second information and determines the image frame associated with the second information and the image frame associated with the three-dimensional model of the target object in the first video stream; based on the image frame associated with the second information and the image frame associated with the three-dimensional model of the target object, the second information, the three-dimensional model of the target object and the first video stream are fused to obtain the second video stream, and the second video stream is sent to the client so that the viewer can perform corresponding operations on the three-dimensional model of the target object on the playback interface of the second video stream.

[0069] The second piece of information can be understood as the real-time audio or text commentary provided by an intelligent broadcasting system or a human director during the live broadcast of the match, focusing on the first video stream or target events (such as goals, key actions, etc.). This second piece of information can include descriptions of player actions, tactical analysis, background information, etc., aiming to enhance the audience's understanding and interactive experience of the match. For example, in a football match, when a player scores a brilliant goal, the second piece of information could be, "In the 60th minute of this match, striker Zhang San scored a one-on-one goal, demonstrating exceptional individual dribbling skills."

[0070] The second video stream can be understood as a comprehensive video stream that overlays a 3D model of the target object and second information onto the first video stream. The 3D model of the target object can realistically reproduce the player's movements and postures and supports various interactive operations. The 3D model of the target object can be rendered or driven through the 3D posture parameters of the target object and the first control command, achieving a high degree of synchronization with the actual game scene. For example, in a basketball game, when a player completes a spectacular dunk, the electronic device can generate a 3D model of the target object (player and basketball) and embed the 3D model of the target object into the first video stream for users to view from multiple angles.

[0071] The display method refers to how the 3D model of the target object is displayed on the playback interface of the second video stream, including picture-in-picture (PiP) display, split-screen display, etc. Picture-in-picture display allows the 3D model of the target object to be nested in a small window within the main video stream, making it convenient for users to simultaneously watch the live match and the 3D model of the target object. Split-screen display divides the screen into multiple areas, displaying different perspectives or types of video content, improving readability and contrast. Picture-in-picture and split-screen display methods can be flexibly switched according to user needs or broadcaster settings to enhance user immersion and interactive experience.

[0072] In this embodiment, a second video stream is generated by embedding a 3D model and second information of the target object into the first video stream. The 3D model of the target object is then displayed in a predefined manner on the playback interface of the second video stream. Viewers can not only see realistic match footage but also gain a deeper understanding of the target object's actions and tactical execution through the 3D model, enhancing their comprehension and participation in the event. Furthermore, viewers can interact with the 3D model, increasing interactivity and engagement, and further improving overall viewing satisfaction and interactive effects.

[0073] To reduce the data processing load on electronic devices and save their processing resources, the 3D model of the target object can be rendered or driven on the terminal side. Based on this, in one embodiment, the method further includes:

[0074] If a target event is detected in the first video stream, the three-dimensional pose parameters of the target object are extracted from the first video stream.

[0075] Sending the first information to the client includes:

[0076] The first video stream, the 3D model of the target object, and the 3D pose parameters of the target object are sent to the client; wherein...

[0077] There is a correlation between the three-dimensional pose parameters of the target object and the image frames in the first video stream in which the target object appears; the three-dimensional pose parameters of the target object are used by the terminal running the client to render the three-dimensional model of the target object in conjunction with the user's second control command for the three-dimensional model of the target object.

[0078] Here, please refer to the relevant description above for how to extract the 3D pose parameters of the target object from the first video stream; it will not be repeated here.

[0079] The correlation between the 3D pose parameters of a target object and the target image frame means that the 3D pose parameters of the target object extracted at a specific time point correspond to a specific image frame in the first video stream that contains the target object. This time synchronization relationship ensures that the 3D model can remain consistent with the original video content during playback, thereby achieving natural animation effects and accurate motion reproduction. For example, if a target object performs a breakthrough action at frame 50, then the 3D pose parameters of the target object should be associated with the image frame at frame 50 so that the 3D model of the target object can accurately reproduce this action.

[0080] The first video stream, the 3D model of the target object, and the 3D pose parameters of the target object sent by the electronic device to the client are decoupled. This allows the terminal running the client to render or drive the 3D model of the target object based on the 3D pose parameters and the second control parameters after receiving the user's second control command for the target object, thereby reproducing the dynamic effect of the target object associated with the target event in the first video stream. Here, "user" refers to an ordinary viewer or other non-broadcast user using the client. The second control command can be partially the same as or completely different from the first control command; no restrictions are placed here. For example, the second control command may include one or more of the following: 360-degree multi-angle viewing, rotating the viewpoint, slowing down, speeding up, moving, zooming in, zooming out, etc. Users can control the display method of the 3D model of the target object through interactive operations (such as swiping, zooming, rotating, etc.), thereby providing users with a more personalized viewing experience. For example, in picture-in-picture mode, users can adjust the position of the 3D model of the target object, zoom in on the hand movements of the target object through gestures, or rotate the 3D model of the target object through gestures to view the details of the 3D model of the target object from different angles.

[0081] In practical applications, when a user clicks on a control on the playback interface of the first video stream, the terminal running the client can generate a corresponding second control command based on the user's action. Then, based on the second control command and the target object's 3D pose parameters, the rendering state of the target object's 3D model is updated. For example, a user can choose to watch a slow-motion replay of the target object and rotate its 3D model using gestures to observe its technical movements from a 360-degree perspective, helping the user to better understand the competition process.

[0082] In this embodiment, by extracting the 3D pose parameters of the target object when a target event is detected, and sending the 3D pose parameters and the 3D model of the target object to the client, the terminal device can render or drive the 3D model of the target object based on the 3D pose parameters and a second control command for the 3D model of the target object. This not only helps users understand the competition process more deeply, but also enhances interactivity and participation, thereby improving the user's immersion and enjoyment in watching the game, and ultimately supporting more diverse competition teaching, tactical analysis, and interactive scenarios.

[0083] In any of the above embodiments, extracting the three-dimensional pose parameters of the target object in the first video stream includes:

[0084] In the first video stream, two-dimensional images of the target object from multiple perspectives and camera parameters of the two-dimensional images are obtained;

[0085] Based on the two-dimensional images of the target object from multiple viewpoints and the camera parameters of the two-dimensional images, the three-dimensional pose parameters of the target object are determined.

[0086] Here, since the image frames in the first video stream carry corresponding camera parameters, after the electronic device acquires two-dimensional images of the target object from multiple viewpoints in the first video stream, it obtains the corresponding camera parameters from the image frames containing the two-dimensional images of the target object from multiple viewpoints, thereby obtaining the camera parameters of the two-dimensional images of the target object from multiple viewpoints. For any target object, based on the two-dimensional images of the target object from multiple viewpoints and the camera parameters of the two-dimensional images, the three-dimensional pose parameters of the target object are determined. For example, based on the two-dimensional images of the target object from multiple viewpoints, two-dimensional keypoints of the target object in each viewpoint are determined; based on a multi-view geometric algorithm, multi-view matching is performed on the two-dimensional keypoints of the target object in each viewpoint to obtain the matching results; based on the matching results and the camera parameters of the two-dimensional images of the target object from multiple viewpoints, the three-dimensional pose parameters of the target object are determined.

[0087] In this context, two-dimensional images of the target object from multiple perspectives can be understood as planar image data of the target object captured from different angles. Camera parameters refer to the geometric and optical attribute information recorded by each camera device during shooting, used to describe key parameters such as the camera's position, orientation, and focal length in space. Camera parameters include, but are not limited to, intrinsic and / or extrinsic parameters. Intrinsic parameters include, but are not limited to, one or more of the following: aperture, focal length, depth of field, shutter speed, ISO, distortion coefficient, and white balance. Extrinsic parameters include, but are not limited to, transformation matrices and / or translation vectors.

[0088] In practical applications, such as in live sports broadcasts, multiple cameras arranged in a circular pattern can be deployed to ensure that the target object's movements can be captured from multiple angles. Each camera can record its camera parameters and associate them with the image frames captured by the camera to obtain the corresponding video stream. The association between the camera parameters and the image frames captured by the camera allows for the accurate reconstruction of the target object's spatial position and posture using the acquired camera parameters.

[0089] By acquiring camera parameters and 2D images of the target object from multiple perspectives, the system can obtain a more comprehensive dataset. This dataset helps improve the accuracy of calculating the 3D pose parameters of the target object. Employing a multi-view collaborative acquisition method based on camera parameters and 2D images of the target object from multiple perspectives can effectively reduce the impact of occlusion problems, thereby resulting in a more realistic and accurate 3D model.

[0090] In this embodiment, the electronic device can use two-dimensional images of the target object from multiple perspectives and the camera parameters of the two-dimensional images, combined with a multi-view geometric algorithm, to calculate the three-dimensional pose parameters of the target object. This can effectively solve the problem of information loss caused by occlusion in a single perspective, thereby improving the accuracy and stability of the three-dimensional pose parameters of the target object, and thus achieving a more realistic three-dimensional modeling and model-driven effect.

[0091] The following section provides a more detailed description of this application with reference to application examples.

[0092] like Figure 3As shown, in scenarios such as tactical analysis, match instruction, and match commentary, electronic devices can acquire multiple video streams captured from different perspectives or shooting angles by multiple cameras set up around the match venue. These video streams are then stitched together and transcoded to obtain a multi-view video stream, i.e., the first video stream. The system then identifies highlights and key players related to those highlights within the first video stream; models these key players to obtain their 3D models; extracts the 3D posture parameters of the key players from the first video stream; and, based on these 3D posture parameters and the first control command triggered by the anchor or commentator for the 3D model of the key players, controls the key players... The electronic device renders or drives the 3D model of the key player, and merges the second information, the rendered or driven 3D model, and the first video stream to obtain a second video stream. This second video stream is then sent to the terminal so that the terminal displays the 3D model of the key player when playing highlights from the first video stream. Alternatively, the electronic device sends the first video stream, the 3D model of the key player, and the 3D pose parameters of the key player to the terminal. This allows the terminal to render or drive the 3D model of the key player based on the 3D pose parameters and the second control command when it receives a second control command triggered by a viewer regarding the 3D model of the key player during the playback of the first video stream. The specific steps are as follows:

[0093] Step 1: Obtain the first video stream, which is obtained from multiple perspectives of the same location.

[0094] Here, as Figure 2 As shown, it is possible to acquire video streams from multiple perspectives captured by a 360-degree surround live streaming camera around the competition venue or event venue, and then stitch and transcode the video streams from multiple perspectives to obtain the first video stream.

[0095] Step 2: Detect highlights, i.e. target events, such as goals, in the first video stream; if highlights are detected, identify the key players associated with the highlights, i.e. target objects.

[0096] Here, after obtaining the first video stream, highlights can be designated by a human director or identified by an intelligent directing system; therefore, as Figure 4 As shown, electronic devices can capture highlights specified by a human director, or they can use an intelligent broadcasting system to identify highlights in the first video stream by tracking the ball's state in real time. After identifying the highlights, key players associated with those highlights can be determined based on the content related to them in the first video stream. For example, key players can be identified by the intelligent broadcasting system based on the relationship between the ball and the players, or key players can be acquired by a human director based on scene requirements. Figure 5As shown. Key players include, but are not limited to, the players who score goals.

[0097] In practical applications, electronic devices can analyze the first video stream, track the ball's state, and automatically identify highlight moments upon detecting a goal, generating a short video clip of the moment (e.g., 15 seconds) up to the goal. The device can then identify the player who scored the goal within this short video clip. Alternatively, the electronic device can acquire short video clips of highlight moments directly specified by a human director, identifying the goal scorer within these clips, or directly acquire key players marked by the human director. These highlight video clips can be understood as multi-view highlight videos.

[0098] Step 3: If a highlight moment (target event) is detected in the first video stream, generate a 3D model of the key player associated with the highlight moment.

[0099] Here, when the electronic device detects a highlight moment in the first video stream, it extracts the feature information of the key player associated with that highlight moment from the first video stream. Based on the feature information of the key player, it performs a 3D modeling of the key player to obtain a 3D model of the key player. Subsequently, it can also extract the 2D image features of the key player from multiple perspectives in the first video stream; based on the 2D image features of the key player from multiple perspectives, it adjusts the 3D model of the key player.

[0100] In practical applications, electronic devices can use deep learning or other image processing techniques to extract key player features such as contours and bounding boxes from the first video stream; combine them with parametric human models (such as SMPL, SCAPE, etc.) to initialize the key player's 3D model based on the key player's feature information; and then iteratively adjust the model parameters of the initialized key player's 3D model through optimization algorithms (such as gradient descent, least squares, etc.) to minimize the error between the key player's 3D model and the key player's 2D image features.

[0101] Furthermore, the electronic device can acquire two-dimensional images of the key player from different perspectives in the first video stream; based on the two-dimensional images of the key player from multiple perspectives, determine the feature information of the key player from each perspective; perform collaborative analysis on the feature information of the key player from each perspective to obtain the complete outline and facial image of the key player, and construct a three-dimensional model of the key player based on the complete outline and facial image of the key player. By performing multi-view collaborative analysis on the two-dimensional images of the key player from multiple perspectives, the occlusion problem can be solved by comprehensively utilizing the two-dimensional image information from multiple perspectives, forming a complete description of the key player from multiple perspectives.

[0102] Step 4: Extract the 3D pose parameters of key players from the first video stream.

[0103] In one embodiment, extracting the three-dimensional pose parameters of key players from the first video stream includes:

[0104] In the first video stream, acquire 2D images of key players from multiple perspectives and their camera parameters; based on the 2D images of key players from multiple perspectives and their camera parameters, determine the 3D pose parameters of key players.

[0105] Specifically, two-dimensional images of key players from different perspectives can be acquired in the first video stream. Based on the two-dimensional images of key players from multiple perspectives, two-dimensional key points of key players from each perspective can be determined. Based on a multi-view geometric algorithm, multi-view matching is performed on the two-dimensional key points of the target object key players from each perspective to obtain the matching results. Based on the matching results and the camera parameters of the two-dimensional images of key players from multiple perspectives, the three-dimensional pose parameters of key players can be determined.

[0106] It's important to note that collaborative analysis of key players' 2D images from multiple perspectives is crucial for obtaining their feature information from these different viewpoints. This comprehensive utilization of feature information from multiple perspectives helps address occlusion issues and yields the player's complete silhouette. For instance, when a key player is occluded from a particular viewpoint, techniques such as deep learning and image matching can be used to analyze and infer the position and pose of the occluded portion using 2D images of the player from other perspectives. In complex real-world environments (such as multi-person interactions or object occlusion), multi-view video offers the advantage over monocular video by providing video information from multiple perspectives of the same object, enabling more comprehensive analysis.

[0107] After performing step 4, the electronic device may perform steps 5 to 6, or perform step 7.

[0108] Step 5: Obtain the first control command for the 3D model of the key player; based on the 3D posture parameters of the key player and the first control command, render the 3D model of the key player to obtain the rendered 3D model of the key player.

[0109] Here, the first control command for the 3D model of key players can be obtained from the local configuration, or it can be a control command triggered by the anchor or commentator on the playback interface of the first video stream, or it can be identified from the commentary content of the first video stream or highlights.

[0110] Based on the key player's 3D posture parameters and the first control command, the key player's 3D model is rendered or driven to present a movement effect similar to that of a real player; for the specific implementation process, please refer to the relevant description above, which will not be repeated here.

[0111] Step 6: Obtain the second information, which includes the commentary content of the first video stream or the commentary content of the highlights; based on the image frames associated with the second information in the first video stream, and the image frames associated with the 3D model of the key player in the first video stream, fuse the second information, the 3D model of the key player, and the first video stream to obtain the second video stream; send the second video stream to the terminal.

[0112] Here, the second information can be tactical analysis information or video commentary from the first video stream. The 3D model of the key player is displayed in a customized manner on the playback interface of the second video stream, including picture-in-picture or split-screen display. For example, when the terminal receives the second video stream, it can display the rendered 3D model of the key player in picture-in-picture or split-screen mode while playing a highlight moment included in the second video stream, allowing the user to perform corresponding operations, such as moving the position of the key player's 3D model, zooming in or out, etc.

[0113] It should be noted that the 3D model of the key player, which is driven or rendered based on the 3D posture parameters of the key player and the first control command, supports 360-degree display in combination with tactical analysis; and supports display in picture-in-picture or split-screen mode according to the requirements of the anchor. It also supports 360-degree multi-angle viewing, slow motion, pause and other methods to display and explain the actions and skills of the key player's 3D model.

[0114] Among them, such as Figure 6 As shown, in a scenario where key players' 3D models are displayed in a picture-in-picture format, the broadcaster can specify the position of the key players' 3D models within the video playback interface, such as the bottom left corner or the top right corner. Figure 7 As shown, in a scenario where the 3D models of key players are displayed in a split-screen manner, the anchor can flexibly choose the background color of the 3D models of key players, such as white, black, or blue, based on the key players' clothing and other elements.

[0115] Step 7: Send the first video stream, the 3D model of the key player, and the 3D pose parameters of the key player to the terminal.

[0116] Among them, there is a correlation between the three-dimensional pose parameters of the key player and the image frames in the first video stream in which the key player appears; the three-dimensional pose parameters of the key player are used by the terminal to render or drive the three-dimensional model of the key player in combination with the user's second control command for the three-dimensional model of the key player.

[0117] Here, the control over the 3D model of the key player can be given to the user, i.e., the viewer; the user controls the interaction between themselves and the 3D model of the key player to trigger a second control command on the 3D model of the key player in the playback interface of the first video stream. The second control command includes one or more of the following:

[0118] Users can move the position of the 3D models of key players in picture-in-picture mode;

[0119] Users can zoom in and out on the 3D models of key players to observe specific areas in more detail;

[0120] Users can rotate the 3D model of a key player to view it in 360 degrees by touching the screen or using gestures.

[0121] When the terminal receives a second control command for the 3D model of the key player, it can render or drive the 3D model of the key player based on the second control command for the 3D model of the key player and the 3D posture parameters of the key player, so as to obtain the rendered or driven 3D model, so that the user can view the 3D model of the key player on the playback interface of the first video stream.

[0122] Specifically, the terminal can display a 3D model of a key player associated with a highlight moment on the playback interface at the moment or time period when the video stream is playing. This allows the user to trigger a second control command for the 3D model of the key player on the playback interface of the first video stream. Based on the second control command for the 3D model of the key player and the 3D posture parameters of the key player, the terminal can render or drive the 3D model of the key player.

[0123] The above solution can provide users with a digital-real integrated viewing or teaching experience, enhancing user participation and enjoyment. It can be applied to scenarios such as event commentary, tactical analysis, and event teaching, as well as interactive viewing scenarios for spectators.

[0124] To implement the image processing method of this application embodiment, this application embodiment also provides an image processing apparatus, which is installed on an electronic device or server, such as... Figure 8 As shown, the device includes:

[0125] The first acquisition unit 801 is used to acquire a first video stream, which is obtained from multiple perspectives of the same site.

[0126] Modeling unit 802 is used to generate a three-dimensional model of the target object associated with the target event if a target event is detected in the first video stream;

[0127] The sending unit 803 is used to send first information to the client; wherein the first information includes at least the first video stream and a three-dimensional model of the target object, and the three-dimensional model of the target object is used to display on the playback interface of the first video stream for the user to perform corresponding operations.

[0128] In one embodiment, the modeling unit 802 is specifically configured to: if a target event is detected in the first video stream, extract feature information of the target object associated with the target event in the first video stream; and perform three-dimensional modeling of the target object based on the feature information of the target object to obtain a three-dimensional model of the target object.

[0129] In one embodiment, the modeling unit 802 is further configured to extract the three-dimensional pose parameters of the target object from the first video stream, and obtain a first control command for the three-dimensional model of the target object; and to render the three-dimensional model of the target object based on the three-dimensional pose parameters of the target object and the first control command to obtain the rendered three-dimensional model of the target object.

[0130] In one embodiment, the device further includes:

[0131] The second acquisition unit is used to extract two-dimensional image features of the target object from multiple perspectives in the first video stream;

[0132] An adjustment unit is used to adjust the three-dimensional model of the target object based on the two-dimensional image features of the target object from the multiple viewpoints.

[0133] In one embodiment, the device further includes:

[0134] The third acquisition unit is used to acquire second information, the second information including the narration content of the first video stream or the narration content of the target event;

[0135] The fusion unit is used to fuse the second information, the three-dimensional model of the target object and the first video stream based on the image frames associated with the second information in the first video stream and the image frames associated with the three-dimensional model of the target object in the first video stream to obtain a second video stream. The three-dimensional model of the target object is used to be displayed in a set manner on the playback interface of the second video stream.

[0136] The sending unit 803 is specifically used to send the second video stream to the client.

[0137] In one embodiment, the device further includes:

[0138] The fourth acquisition unit is used to extract the three-dimensional pose parameters of the target object in the first video stream if a target event is detected in the first video stream.

[0139] The sending unit 803 is specifically used to send the first video stream, the three-dimensional model of the target object, and the three-dimensional pose parameters of the target object to the client; wherein, there is a correlation between the three-dimensional pose parameters of the target object and the image frames in the first video stream in which the target object appears; the three-dimensional pose parameters of the target object are used by the terminal running the client to render the three-dimensional model of the target object in conjunction with the user's second control command for the three-dimensional model of the target object.

[0140] In one embodiment, the modeling unit 802 or the fourth acquisition unit is specifically used to acquire, in the first video stream, two-dimensional images of the target object from multiple perspectives and camera parameters of the two-dimensional images; and to determine the three-dimensional pose parameters of the target object based on the two-dimensional images of the target object from the multiple perspectives and camera parameters of the two-dimensional images.

[0141] In practical applications, the first acquisition unit 801, the modeling unit 802, the second acquisition unit, the adjustment unit, the third acquisition unit, the fusion unit, and the fourth acquisition unit can be implemented by a processor in an image processing device, and the sending unit 803 can be implemented by a processor in an image processing device combined with a communication interface.

[0142] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the image processing apparatus and the image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0143] Based on the hardware implementation of the above program modules, and in order to implement the method on the electronic device side of the embodiments of this application, the embodiments of this application also provide an electronic device, such as... Figure 9 As shown, the electronic device 900 includes:

[0144] The communication interface 901 enables information exchange with other network nodes;

[0145] The processor 902 is connected to the communication interface 901 to enable information interaction with other network nodes and to execute the methods provided by one or more of the above-described technical solutions when running a computer program. The computer program is stored in the memory 903.

[0146] Specifically, the processor 902 is configured to acquire a first video stream, which is captured from multiple perspectives of the same location; and to generate a three-dimensional model of a target object associated with the target event if a target event is detected in the first video stream.

[0147] The communication interface 901 is used to send first information to the client; wherein the first information includes at least the first video stream and a three-dimensional model of the target object, and the three-dimensional model of the target object is used to display on the playback interface of the first video stream for the user to perform corresponding operations.

[0148] In one embodiment, the processor 902 is specifically configured to: if a target event is detected in the first video stream, extract feature information of a target object associated with the target event in the first video stream; and perform three-dimensional modeling of the target object based on the feature information of the target object to obtain a three-dimensional model of the target object.

[0149] In one embodiment, the processor 902 is further configured to extract three-dimensional pose parameters of the target object from the first video stream, and obtain a first control instruction for the three-dimensional model of the target object; and to render the three-dimensional model of the target object based on the three-dimensional pose parameters of the target object and the first control instruction to obtain the rendered three-dimensional model of the target object.

[0150] In one embodiment, the processor 902 is further configured to extract two-dimensional image features of the target object from multiple viewpoints in the first video stream; and to adjust the three-dimensional model of the target object based on the two-dimensional image features of the target object from the multiple viewpoints.

[0151] In one embodiment, the processor 902 is further configured to acquire second information, the second information including the narration content of the first video stream or the narration content of the target event; and to fuse the second information, the three-dimensional model of the target object and the first video stream based on the image frames associated with the second information in the first video stream and the image frames associated with the three-dimensional model of the target object in the first video stream to obtain a second video stream, wherein the three-dimensional model of the target object is used to display in a set manner on the playback interface of the second video stream;

[0152] The communication interface 901 is specifically used to send the second video stream to the client.

[0153] In one embodiment, the processor 902 is further configured to extract three-dimensional pose parameters of the target object in the first video stream if a target event is detected in the first video stream;

[0154] The communication interface 901 is specifically used to send the first video stream, the three-dimensional model of the target object, and the three-dimensional pose parameters of the target object to the client; wherein, there is a correlation between the three-dimensional pose parameters of the target object and the image frames in the first video stream in which the target object appears; the three-dimensional pose parameters of the target object are used by the terminal running the client to render the three-dimensional model of the target object in conjunction with the user's second control command for the three-dimensional model of the target object.

[0155] In one embodiment, the processor 902 is specifically configured to acquire, in the first video stream, two-dimensional images of the target object from multiple perspectives and camera parameters of the two-dimensional images; and determine the three-dimensional pose parameters of the target object based on the two-dimensional images of the target object from the multiple perspectives and the camera parameters of the two-dimensional images.

[0156] It should be noted that the specific processing procedures of processor 902 and communication interface 901 can be understood by referring to the above method.

[0157] Of course, in practical applications, the various components in electronic device 900 are coupled together through bus system 904. It can be understood that bus system 904 is used to realize the connection and communication between these components. In addition to a data bus, bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 9 The general designated all buses as Bus System 904.

[0158] The memory 903 in this embodiment is used to store various types of data to support the operation of the electronic device 900. Examples of such data include any computer program used to operate on the electronic device 900.

[0159] The methods disclosed in the embodiments of this application can be applied to, or implemented by, the processor 902. The processor 902 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware or by instructions in software form within the processor 902. The processor 902 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 902 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically a memory 903. The processor 902 reads information from the memory 903 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0160] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0161] It is understood that the memory (memory 903) in this embodiment of the application can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0162] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 903 storing a computer program, which can be executed by the processor 902 of the electronic device 900 to complete the steps described in the aforementioned electronic device-side method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0163] For example, this application also provides a computer program product, including a computer program that can be executed by a processor 902 of an electronic device 900 to complete the steps described in the aforementioned electronic device-side method.

[0164] It should be noted that terms such as "first" and "second" are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. "Multiple" can refer to two or more items, and "multiple" can refer to two or more items. The term "and / or" in this document merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "one or more" in this document refers to any combination of at least two of the multiple elements. For example, including one or more of A, B, and C can represent including any one or at least two or more elements selected from the set consisting of A, B, and C.

[0165] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0166] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a first video stream, the first video stream being obtained by shooting a same site from multiple perspectives; if a target event is detected in the first video stream, generating a three-dimensional model of a target object associated with the target event; sending first information to a client, wherein the first information at least comprises the first video stream and the three-dimensional model of the target object, and the three-dimensional model of the target object is used to be displayed on a playing interface of the first video stream for a user to perform corresponding operations.

2. The method of claim 1, wherein, If the target event is detected in the first video stream, the three-dimensional model of the target object associated with the target event is generated, comprising: if a target event is detected in the first video stream, extracting feature information of a target object associated with the target event in the first video stream; based on the feature information of the target object, performing three-dimensional modeling on the target object to obtain a three-dimensional model of the target object.

3. The method of claim 2, wherein, If the target event is detected in the first video stream, the three-dimensional model of the target object associated with the target event is generated, further comprising: extracting three-dimensional pose parameters of the target object in the first video stream, and acquiring a first control instruction for the three-dimensional model of the target object; based on the three-dimensional pose parameters of the target object and the first control instruction, rendering the three-dimensional model of the target object to obtain a three-dimensional model of the target object after rendering.

4. The method of claim 2, wherein, The method further comprises: extracting two-dimensional image features of the target object under multiple perspectives in the first video stream; based on the two-dimensional image features of the target object under the multiple perspectives, adjusting the three-dimensional model of the target object.

5. The method according to any one of claims 1 to 4, characterized in that, The first information is sent to the client, comprising: acquiring second information, the second information comprising commentary content of the first video stream or commentary content of the target event; based on image frames associated with the second information in the first video stream and image frames associated with the three-dimensional model of the target object in the first video stream, fusing the second information, the three-dimensional model of the target object and the first video stream to obtain a second video stream, the three-dimensional model of the target object being used to be displayed in a set manner on a playing interface of the second video stream; sending the second video stream to the client.

6. The method of claim 1, 2, or 4, wherein, The method further comprises: if a target event is detected in the first video stream, extracting three-dimensional pose parameters of the target object in the first video stream; The first information is sent to the client, comprising: sending the first video stream, the three-dimensional model of the target object and the three-dimensional pose parameters of the target object to the client; wherein, the three-dimensional pose parameters of the target object and image frames in which the target object appears in the first video stream have an associated relationship; the three-dimensional pose parameters of the target object are used to render the three-dimensional model of the target object in combination with a second control instruction of a user for the three-dimensional model of the target object by a terminal running the client.

7. The method of claim 6, wherein, The three-dimensional pose parameters of the target object are extracted in the first video stream, comprising: In the first video stream, a two-dimensional image and a camera parameter of the two-dimensional image of the target object under multiple perspectives are acquired; Based on the two-dimensional image and the camera parameter of the two-dimensional image of the target object under the multiple perspectives, a three-dimensional pose parameter of the target object is determined.

8. An electronic device, comprising: A computer program product comprising a computer readable medium storing instructions which, when executed by a computer, cause the computer to perform the steps of any of claims 1 to 7. The computer program product, when executed by the processor, implements the steps of the method of any of claims 1 to 7.

9. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, implements the steps of the method of any of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any of claims 1 to 7.

Images