Target object display method, electronic device, storage medium, and program product

By leveraging efficient network connectivity between the client and the cloud, and AI detection technology, the system automatically selects the appropriate camera to track the target object, resolving the issue of unsmooth perspective switching in multi-camera systems and achieving smooth video playback and ease of operation.

CN123123714APending Publication Date: 2026-11-17MIGU CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610976643.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-01
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, multi-camera systems rely on active user operation and lack intelligent understanding and response, resulting in less smooth video playback, especially in rapidly changing scenes where key shots are easily missed.

Method used

By establishing an efficient network connection between the client and the cloud, and utilizing multi-camera surround setup and AI detection technology, the system can track target objects in real time and automatically select the appropriate camera to keep the target object in the preset position on the video playback interface, reducing manual switching operations.

Benefits of technology

It improves the smoothness of video playback, reduces the operational burden, provides a seamless viewing experience, and ensures that the target object is always displayed in the preset position.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123714A_ABST
    Figure CN123123714A_ABST
Patent Text Reader

Abstract

This application provides a method and system for displaying a target object. The method includes: in response to an instruction operation on a target object within a target area displayed on a video playback interface, sending an object detection request to the cloud; wherein the video playback interface displays footage captured by any one of a plurality of preset cameras, which are arranged around the target area and capture images of the target area; the cloud performs target tracking on the target object based on the object detection request and periodically sends target tracking information to the client; based on the target tracking information sent by the cloud, a target camera is determined from the plurality of preset cameras; and the target video frame captured by the target camera is displayed on the video playback interface, wherein the target object is located at a preset position in the target video frame. This method can improve the smoothness of video playback during target object display.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to video processing technology, and more particularly to a method for displaying a target object, an electronic device, a storage medium, and a program product. Background Technology

[0002] Videos in specific scenarios often contain objects that require special attention. By tracking specific objects in a video of a particular scenario and displaying them in the center of the viewer's field of vision, the user's viewing experience can be improved. For example, by tracking specific athletes in sports event videos and ensuring that the athlete is displayed in the center of the frame, the user's best moments can be reduced by minimizing the loss of those athletes' highlights.

[0003] Currently, since a single camera's perspective cannot cover all the scenes in a specific scenario, a 360° camera system consisting of multiple cameras is typically used to monitor the scene from all angles. Users can switch between different camera perspectives via touchscreen or mouse dragging to track objects they want to focus on in real time, allowing the objects to be displayed at specific locations on the screen and providing a certain degree of viewing freedom.

[0004] However, this object display method has an unsmooth viewpoint switching process, which in turn makes the video playback process unsmooth as well. Summary of the Invention

[0005] This application provides a method, system, electronic device, computer-readable storage medium, and computer program product for displaying a target object, which can improve the smoothness of video playback during the display of the target object.

[0006] In a first aspect, embodiments of this application provide a method for displaying a target object, applied to a client, the method comprising: In response to an instruction operation on a target object within a target area displayed on the video playback interface, an object detection request is sent to the cloud. The video playback interface displays footage captured by any one of a set of preset cameras, which are arranged around the target area and capture images of it. The cloud tracks the target object based on the object detection request and periodically sends target tracking information to the client. Based on the target tracking information sent from the cloud, the target camera is determined from multiple preset cameras; The video playback interface displays target video frames captured by the target camera, in which the target object is located at a preset position.

[0007] In the above scheme, the target tracking information includes the target position information of the target object in the image captured by the first camera; the first camera is the camera corresponding to the image currently displayed on the video playback interface; based on the target tracking information sent from the cloud, the target camera is determined from a set of preset cameras, including: Based on the target location information and the preset first direction angle of the first camera, the panoramic direction angle of the target object is determined; the panoramic direction angle is the direction angle of the target object in the panoramic view; the panoramic view is generated by images captured by multiple preset cameras respectively; The target camera is determined from a set of preset cameras based on the panoramic orientation angle.

[0008] In the above scheme, determining the panoramic orientation angle of the target object based on the target location information and the preset first orientation angle of the first camera includes: Based on the target location information, determine the position offset ratio of the target object relative to the center of the image captured by the first camera; Based on the position offset ratio, determine the angular offset of the target object relative to the center of the image captured by the first camera; The panoramic orientation angle of the target object is determined based on the angular offset and the preset first orientation angle of the first camera.

[0009] In the above scheme, in response to an instruction operation targeting a target object within a target area displayed on the video playback interface, an object detection request is sent to the cloud, including: In response to an instruction operation on a target object within a target area displayed on the video playback interface, the timestamp information of the instruction operation, the instruction location information corresponding to the instruction operation, and the camera identifier of the second camera are determined; the second camera is the camera corresponding to the image indicated by the instruction operation. Based on the indicated location information, timestamp information, and camera identifier of the second camera, generate an object detection request corresponding to the indicated operation; Send an object detection request to the cloud.

[0010] In the above scheme, determining the indication location information corresponding to the indication operation includes: Obtain the indication direction vector corresponding to the indication operation; Determine the yaw angle of the indicated direction vector relative to the preset horizontal plane; Based on the yaw angle and the preset second direction angle of the second camera, determine the offset angle of the indication direction vector relative to the center of the screen corresponding to the indication operation; If the offset angle indicates that the instruction operation is within the field of view of the second camera, then the instruction position information of the instruction operation is determined based on the offset angle.

[0011] The above scheme also includes the following methods: If the difference in azimuth between the target camera and the first camera is greater than a preset threshold, at least one transition camera between the first camera and the target camera is determined. The video playback interface sequentially switches from historical video frames captured by the first camera to transitional video frames captured by at least one transitional camera, and then switches from at least one transitional video frame to the target video frame captured by the target camera. This application also provides another method for displaying a target object, applied in the cloud, the method comprising: Obtain the object detection request sent by the client, wherein the object detection request is generated according to the instruction operation; the instruction operation is the instruction operation for the target object in the target area within the screen displayed on the video playback interface; the video playback interface displays the screen captured by any one of the preset multiple cameras, which are arranged around the target area and capture images of the target area. The system tracks the target object and periodically sends target tracking information to the client. The target tracking information instructs the client to select the target camera from multiple preset cameras and displays the target video frames captured by the target camera on the video playback interface.

[0012] In the above scheme, the object detection request carries the indication location information corresponding to the indication operation, the timestamp information of the indication operation triggering, and the camera identifier of the second camera; the second camera is the camera corresponding to the image indicated by the indication operation; target tracking of the target object includes: Within the first cycle, the first video frame of the target object captured by the second camera is determined based on the timestamp information and the camera identifier; Based on the indicated location information, determine the first location information of the target object in the first video frame; Based on the first location information and the object identifier of the target object, target tracking information of the target object is generated.

[0013] In the above scheme, determining the first position information of the target object in the first video frame based on the indicated position information includes: Determine the first detection bounding boxes of multiple objects in the first video frame and the confidence level of each first detection bounding box; Determine the matching degree between the indicated location information and each first detection box; Based on the matching degree and confidence degree of each first detection box, determine the second detection box corresponding to the target object from multiple first detection boxes; Based on the second detection box, the first position information of the target object in the first video frame is determined.

[0014] The above scheme includes target tracking of the target object, including: Within the Nth period, determine the second video frame captured by the first camera; the first camera is the camera corresponding to the current image displayed on the video playback interface; N is a positive integer not equal to 1; Obtain historical tracking status information of the target object; Based on historical tracking status information, determine the second position information of the target object in the second video frame; Based on the second location information and the object identifier of the target object, the target tracking information of the target object is determined.

[0015] In the above scheme, the second location information of the target object in the second video frame is determined based on historical tracking status information, including: Based on historical tracking status information, predict the third location information of the target object in the second video frame; Determine the third detection boxes for multiple objects in the second video frame, and the confidence level of each third detection box; Based on the matching degree between the third location information and each third detection box, and the confidence degree corresponding to each third detection box, the fourth detection box of the target object is determined from multiple third detection boxes; Based on the fourth detection box, the second position information of the target object in the second video frame is determined.

[0016] This application provides a target object display system, the system including a client and a cloud, the system including: The client is used to send an object detection request to the cloud in response to an instruction operation on a target object in a target area displayed on the video playback interface; wherein, the video playback interface displays the image captured by any one of a set of preset cameras, and the preset cameras are arranged around the target area to capture the target area. In the cloud, it is used to track target objects and periodically send target tracking information to the client; The client is used to determine the target camera from multiple preset cameras based on target tracking information sent from the cloud; and to display the target video frames captured by the target camera on the video playback interface, wherein the target object in the target video frame is located at a preset position.

[0017] This application provides an electronic device, which includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in memory, implements the methods provided in the embodiments of this application.

[0018] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the target object method provided in this application when executed by a processor.

[0019] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the target object method provided in this application.

[0020] In this embodiment, the client, in response to an instruction to target an object within any frame of the video playback interface, sends an object detection request to the cloud. The video playback interface displays footage captured by any one of a plurality of preset cameras, which are arranged around the target area and capture images of it. The cloud tracks the target object based on the object detection request and periodically sends target tracking information to the client. The client, based on the target tracking information sent by the cloud, determines the target camera from the plurality of preset cameras. The target video frame captured by the target camera is displayed on the video playback interface, where the target object is positioned at a preset location within the target video frame. Throughout this process, the client instructs the cloud to continuously track the target object and periodically send target tracking information to the client. This allows the client to automatically switch to the target video frame corresponding to the target camera at the appropriate intervals based on the target tracking information. This switching process eliminates the need for frequent manual switching, significantly reducing the operational burden. While maintaining the target object in the preset position on the video playback interface, it also improves the smoothness of video playback. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the target object display system architecture provided in the embodiments of this application; Figure 2 This is a first flowchart illustrating the target object display method provided in this application embodiment; Figure 3 This is a schematic diagram of multiple preset cameras arranged around a target area according to an embodiment of this application; Figure 4 This is a second flowchart illustrating the target object display method provided in this application embodiment; Figure 5 This is a schematic diagram illustrating the playback principle of a full-screen player provided in an embodiment of this application; Figure 6 This is a schematic diagram of a hybrid cloud and client architecture provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0022] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In related technologies, to ensure that the video playback interface displays images centered on a specific object and minimizes the omission of highlights of that specific object, the following methods can generally be used: 1. User-manual drag-and-drop: A 360° camera system consisting of multiple cameras is set up to monitor the scene in all directions. Users can switch between different camera perspectives by dragging with a touch screen or mouse to track the objects they want to focus on in real time. The objects can be displayed in a specific position on the screen, allowing for a certain degree of viewing freedom.

[0025] 2. AI-Assisted Detection: With the development of deep learning, more and more video platforms are using AI models to automatically identify highlight moments in videos, such as goals, dunks, and shots, and display the corresponding frames for these highlights. Commonly used models include object detection models (YOLO (You Only Look Once, a single-stage real-time object detection algorithm), Faster R-CNN (Faster Region-based Convolutional Neural Network), and action recognition models (TSN (Temporal Segment Networks), SlowFast (SlowFast Networks for Video Recognition)).

[0026] However, these 360° camera systems generally rely on user-initiated operation and lack intelligent understanding and response to viewing intentions. For example, in rapidly changing events, viewers need to frequently tap the screen to track specific athletes, easily missing crucial shots. In other words, perspective switching is entirely user-driven, and the system lacks automatic focusing capabilities based on visual semantics, resulting in less-than-smooth video playback. Even when combined with AI-powered intelligent detection technology, it is mostly used for background data analysis and fails to map detection results to the perspective control logic in real time. Therefore, current object display technologies still lack a certain degree of smoothness in video playback.

[0027] To address the shortcomings of the aforementioned related technologies, this application provides a target object display method that can improve the smoothness of video playback during target object display. To illustrate the technical solution of this invention, specific embodiments are described below.

[0028] This application provides a method for displaying a target object, such as... Figure 1 As shown, this method is applied to a target object display system 101. System 101 includes a client 102, a cloud 103, and a multi-camera module 104. The client 102 and cloud 103 communicate bidirectionally via 5G (5th Generation Mobile Communication Technology) / dedicated network. The 5G / dedicated network provides efficient and stable network connectivity, ensuring fast and accurate data transmission between the client and cloud. The multi-camera module includes four cameras, each communicating with either the client 102 or the cloud 103. The four cameras are arranged in a 360° panoramic space around the target area. Each camera has a horizontal field of view of 90°, where the horizontal field of view refers to the angle between the two outermost imaging rays of the camera lens on a horizontal plane. The four preset cameras capture images of the target area, forming a 360° panoramic view of the target area. The client 102 displays a video playback interface, showing real-time images captured by any one of the preset cameras.

[0029] The user makes a gesture indicating a target object within the target area displayed on the video playback interface, for example, by pointing to the target object with their index finger. In response, client 102 sends an object detection request to the cloud. Cloud 103 responds to the object detection request, tracks the target object, and periodically sends target tracking information to client 102. Based on the target tracking information sent by cloud 103, client 102 identifies the target camera from a set of preset cameras 104 and displays the target video frame captured by the target camera on the video playback interface, where the target object is positioned at a preset location within the target video frame.

[0030] Figure 2 A flowchart illustrating a target object display method provided in this application embodiment, taking the method applied to a client as an example, the target object display method provided in this application includes the following steps: S201, in response to an instruction operation on a target object within a target area displayed on the video playback interface, sends an object detection request to the cloud.

[0031] The video playback interface displays footage captured by any one of a set of preset cameras. These preset cameras are arranged around the target area and capture images of the target area. The cloud tracks the target object based on the object detection request and periodically sends the target tracking information to the client.

[0032] The target area can be a stadium, examination room, concert hall, or other area requiring monitoring. Multiple cameras are installed around the target area, and the combination of these cameras should provide a 360° panoramic view. The images captured by these cameras are uploaded to the cloud in real time, allowing the cloud to track the target from the footage. For example... Figure 3 The diagram shows 16 cameras, A1 to A16, arranged around the perimeter of basketball court 302. The dotted lines indicate the shooting direction of the cameras, and the angle between the two dotted lines corresponding to each camera is the directional angle of the corresponding camera. Each camera captures a picture of basketball court 302, and the images captured by all the cameras constitute a 360° panoramic view of the basketball court.

[0033] The azimuth difference between two adjacent cameras in a multi-camera setup can be configured. The azimuth difference between adjacent cameras can be the same or different. For example, when the azimuth difference between two adjacent cameras is configured to a fixed interval of 10°, 36 cameras need to be set up to surround the target area at fixed 10° intervals to observe a 360° panorama of the target area. In this case, the azimuth angle of the first camera is 0°, the azimuth angle of the second camera is 10°, ..., and the azimuth angle of the 36th camera is 250°. Alternatively, the azimuth difference between the first and second cameras can be 10°, and the azimuth difference between the second and third cameras can be 12°; no limitation is made here. For ease of description, the following embodiments of this application all use the example of the same azimuth difference between two adjacent cameras to illustrate this solution.

[0034] Images of the target area captured by multiple preset cameras can be stitched together in sequence to form a single frame. For example, images captured by 36 cameras can be stitched together in sequence to form a large 6x6 image. The image captured by one of the cameras can be displayed in the video playback interface through a special player. At the same time, the video playback interface also supports switching to the current image captured by another camera by clicking, swiping, or dragging the screen.

[0035] The target object can be a specific person, animal, or object. For example, the target object could be an athlete, a basketball, or an animal.

[0036] The instruction operation can be a gesture operation, a head rotation operation, etc. For example, when pointing to a target object in the target area displayed on the video playback interface, the client can obtain the instruction position information corresponding to the gesture operation to determine the target object to be indicated; or when pointing to a target object in the target area displayed on the video playback interface by rotating the head, the client can obtain the rotation angle and direction of the head to determine the target object to be indicated.

[0037] Target tracking information includes at least the target location information and the object identifier of the target object. By obtaining the target tracking information, the location detection result corresponding to a specific object identifier can be obtained.

[0038] In this embodiment of the application, when watching the video playback interface, if the user wants to focus on a target object in the target area of ​​the screen, the user can use gestures or head rotation to indicate the target object in the target area of ​​the screen. At this time, the client can respond to the indication operation in a timely manner, generate an object detection request, and send the object detection request to the cloud.

[0039] Furthermore, based on the object detection request, the cloud determines the target object indicated by the instruction operation, assigns an object identifier to the target object, and continuously tracks the target object corresponding to that object identifier, periodically sending the target object's tracking information to the client. In practical applications, the interval can be configured according to requirements, such as pushing the latest target tracking information of the target object to the client every 100ms.

[0040] In this embodiment of the application, the target object is tracked, and the target position information of the target object can be the position information of the target object in the frame captured by the first camera, or the position information of the target object in the panoramic frame, and there is no limitation here.

[0041] In this embodiment, the client responds to an instruction operation on a target object within a target area displayed on the video playback interface. It obtains the instruction location information corresponding to the instruction operation, the camera identifier corresponding to the screen indicated by the instruction operation, and the timestamp of the instruction operation. Based on the instruction location information, the camera identifier corresponding to the screen indicated by the instruction operation, and the timestamp of the instruction operation, the client generates an object detection request. That is, the object detection request includes at least the instruction location information corresponding to the instruction operation, the camera identifier corresponding to the screen indicated by the instruction operation, and the timestamp of the instruction operation. The cloud determines the first video frame corresponding to the instruction operation based on the timestamp and camera identifier of the instruction operation in the object detection request. Then, based on the instruction location information, it detects the target object and its target location information in the first video frame and assigns a special object identifier to the target object so that it can be continuously tracked in each subsequent cycle, obtaining the target tracking information of the target object in each cycle.

[0042] In this embodiment, the cloud can perform target detection using object detection technologies such as YOLO and Faster R-CNN based on object detection requests to determine the target object indicated by the instruction operation. It continuously tracks the target object corresponding to the selected object identifier, updates the target position information of the target object in real time, and generates target tracking information based on the object identifier and target position information of the target object in each cycle, then sends the target tracking information to the client. In this embodiment, a typical client cannot respond to user instructions. Therefore, to enable the client to respond to user instructions, an instruction device can be set up. This instruction device is connected to the client and allows the user to perform instruction operations on target objects within the target area displayed on the video playback interface. This replaces the manual swiping to switch perspectives on the screen. In practical applications, this instruction device can be a gesture instruction device, a head instruction device, etc. When the instruction device is a gesture instruction device, the user can perform click operations on target objects within the target area displayed on the video playback interface; when the instruction device is a head instruction device, the user can perform head rotation operations on target objects within the target area displayed on the video playback interface.

[0043] S202, based on the target tracking information sent from the cloud, determines the target camera from a set of preset cameras.

[0044] Among them, the target camera refers to the camera that places the target object in the frame at a preset position. Generally speaking, in order to improve the viewing experience, the preset position is the center of the frame. However, the preset position may also be the upper left position, the lower left position, etc., depending on the actual needs.

[0045] In this embodiment of the application, the client periodically receives target tracking information from the cloud and dynamically adjusts the camera angle based on the received target tracking information to ensure that after the video playback interface switches to the target video frame captured by the target camera, the target object is at the preset position of the target video frame.

[0046] In this embodiment, the target tracking information includes at least the target location information of the target object. Based on the target location information, the client automatically selects the most suitable target camera, ensuring that the target object is positioned at a preset location in the video frame captured by the target camera. Furthermore, the client uses a camera selection algorithm to automatically select the most suitable target camera based on the target location information of the target object.

[0047] In one embodiment, the target location information of the target object can be the location information of the target object within the panoramic view. The client uses a camera selection algorithm to automatically select the most suitable target camera based on the location information of the target object within the panoramic view.

[0048] In another embodiment, the target location information of the target object can also be the location information of the target object within the frame captured by the first camera. The client uses a camera selection algorithm to automatically select the most suitable target camera based on the location information of the target object within the frame captured by the first camera.

[0049] At this point, the camera selection algorithm completes the selection of the target camera through the following three perspective transformations: local view → panoramic view → optimal view. Specifically, since the obtained information is the target position information corresponding to the image captured by the first camera, and the target object in this image may not be in the preset position, it is necessary to transform the image displayed in the video playback interface from the local view of the first camera to the panoramic view based on the target position information. Then, based on the panoramic view, the target camera corresponding to the optimal view that places the target object in the preset position of the video frame is determined. At this time, the image displayed in the video playback interface switches from the original image to the target video frame captured by the target camera. For example, if the determined target camera is camera 13, it is necessary to switch to the target video frame with the perspective corresponding to the target camera identifier 13.

[0050] Alternatively, the camera selection algorithm can bypass the intermediate step of panoramic view conversion. It can directly convert the view displayed on the playback interface from the first camera's perspective to the perspective of any other camera based on the target object's location information. Then, it can determine whether the target object's location information within the converted perspective meets the conditions for placing the target object in a preset position. If there is location information that meets the conditions, then the corresponding camera is determined to be the target camera.

[0051] S203 displays the target video frames captured by the target camera on the video playback interface.

[0052] In this context, the target object in the target video frame is positioned at a preset location.

[0053] In this embodiment of the application, after the target camera is determined, the video player can smoothly switch the video stream of the video playback interface and display the target video frame captured by the target camera on the video playback interface, so that the target object is in the preset position of the screen. This switching process is continuous and does not require frequent manual instructions from the user, which reduces the operational burden and provides a seamless viewing experience.

[0054] In this embodiment of the application, when the difference in directional angle between the target camera and the first camera is greater than a preset threshold, at least one transition camera is determined to switch from the first camera to the target camera; in the video playback interface, the historical video frames captured by the first camera are sequentially switched to the transition video frames captured by at least one transition camera, and then switched from at least one transition video frame to the target video frame captured by the target camera.

[0055] The first camera is the camera corresponding to the current image displayed on the video playback interface. The camera's azimuth angle is the camera's horizontal azimuth angle, that is, the horizontal angle at which the camera's optical axis deflects clockwise to the right or counterclockwise to the left relative to the direction directly in front of the device in the horizontal plane.

[0056] Multiple preset cameras are arranged in sequence around the target area. The camera identifiers C∈{0, 1, ..., n} are configured in a clockwise or counterclockwise order, with the horizontal line of the three-dimensional space where the target area is located as the reference. Here, n is the identifier of the last camera. For example, when there are a total of 4 cameras, the camera identifiers C∈{0, 1, 2, 3} are configured in a clockwise order.

[0057] When there are multiple transition cameras between the target camera and the first camera, rather than adjacent cameras, a step-by-step switching operation is often required to make the viewpoint switching smooth. That is, in the video playback interface, the historical video frames captured by the first camera are switched sequentially to the transition video frames captured by at least one transition camera, and then the target video frames captured by the target camera are switched from at least one transition video frame to the target video frame captured by the target camera.

[0058] The method for determining whether there are multiple transition cameras between the target camera and the first camera is as follows: Each camera has a preset azimuth angle, which is related to its installation position. When multiple transition cameras exist between the target camera and the first camera, if the difference in azimuth angle between the target camera and the first camera exceeds a preset threshold, the camera identification of the target camera will be affected. C opt Camera identifier C of the first camera t The relationship between them is: C opt -C t >1, therefore, it can be determined based on the camera identifier of the target camera. C opt Camera identifier C of the first camera t The relationship between the target camera and the first camera, or the difference in directional angle between the target camera and the first camera, determines whether there are multiple transition cameras between the target camera and the first camera.

[0059] In other words, when the angular difference between the target camera and the first camera is greater than a preset threshold or C opt -C t If the value is greater than 1, it indicates that there are multiple transition cameras between the target camera and the first camera, and a step-by-step switching operation needs to be performed.

[0060] For example, when the target camera's camera identifier C opt The first camera is labeled C (number 4). t When the value is 1 and the angular difference between two adjacent cameras is 10°, let the preset threshold be 10°. At this time, the angular difference between the target camera and the first camera is 40°-10°=30°>the preset threshold of 10°. C opt -C t If the value is greater than 1, then at least one transition camera is determined between the first camera and the target camera, and the camera identifiers of the transition cameras are 2 and 3.

[0061] In | C opt -C t If |=1 or the difference in azimuth angle between the target camera and the first camera is less than or equal to a preset threshold, it is determined that there are no multiple transition cameras between the target camera and the first camera, and the historical video frame captured by the first camera can be directly switched to the target video frame captured by the target camera on the video playback interface.

[0062] In this embodiment of the application, a viewpoint switching smoothing coefficient can also be introduced during the video frame switching process. This viewpoint switching smoothing coefficient is used to control the speed of video frame switching. That is, the original video frame switching speed is V1. Based on the viewpoint switching smoothing coefficient a, the video frame switching speed is adjusted to V2=a V1. According to the adjusted video frame switching speed, the video playback interface sequentially switches from the historical video frames captured by the first camera to the transition video frames captured by at least one transition camera, and then switches from at least one transition video frame to the target video frame captured by the target camera to avoid screen jumps.

[0063] In the above method, the client responds to an instruction on a target object within any frame of the video playback interface and sends an object detection request to the cloud. The video playback interface displays footage captured by any one of a set of preset cameras, which are positioned around the target area and capture images of it. The cloud tracks the target object based on the object detection request and periodically sends target tracking information to the client. The client then identifies the target camera from the preset set of cameras based on the target tracking information sent by the cloud. The target video frame captured by the target camera is displayed on the video playback interface, where the target object is positioned at a preset location within the target video frame. Throughout this process, the client instructs the cloud to continuously track the target object and periodically send target tracking information to the client. This allows the client to automatically switch to the target video frame corresponding to the target camera at the appropriate intervals based on the target tracking information. This switching process eliminates the need for frequent manual switching, significantly reducing the operational burden. While maintaining the target object in the preset position on the video playback interface, it also improves the smoothness of video playback.

[0064] In other embodiments of this application, the target tracking information includes the target position information of the target object in the image captured by the first camera; the first camera is the camera corresponding to the image currently displayed on the video playback interface; determining the target camera from a preset plurality of cameras based on the target tracking information sent from the cloud includes: Based on the target location information and the preset first direction angle of the first camera, the panoramic direction angle of the target object is determined; based on the panoramic direction angle, the target camera is determined from a preset set of multiple cameras.

[0065] Among them, the panoramic orientation angle is the orientation angle of the target object in the panoramic view; the panoramic view is generated by capturing images from multiple preset cameras.

[0066] The camera's azimuth angle referred to in this application is the camera's horizontal azimuth angle, that is, the horizontal angle at which the camera's optical axis deflects clockwise to the right or counterclockwise to the left relative to the direction directly in front of the device in the horizontal plane. For example, taking 0° directly in front as a reference, if the target area is set with a horizontal field of view... θ fov With 36 cameras fixed at 10°, forming a 360° panoramic view of the target area, the preset orientation angle of the first camera is 10°, the preset orientation angle of the second camera is 20°, the preset orientation angle of the third camera is 30°, and so on.

[0067] Furthermore, the preset orientation angle of the camera can be based on the horizontal field of view. θ fov In this application, the camera identifier refers to the identifier arranged clockwise / counterclockwise from a reference camera (generally a horizontally oriented camera). The preset orientation angle of the camera can be based on the horizontal field of view. θ fov The product of the camera identifier and the identifier is used to determine the identifier. For example, the camera identifier C of multiple preset cameras. t Let C be the camera identifier of the fourth camera, ∈{0, 1, ..., 35}. t If the value is 4, then the preset orientation angle of the fourth camera is 4 x 10° = 40°.

[0068] In this embodiment, a camera selection algorithm is used to complete the conversion of the following three perspectives: local perspective → panoramic perspective → optimal perspective. Specifically, firstly, a preset first orientation angle of the first camera is obtained. Then, based on the target tracking information and the preset first orientation angle of the first camera, the image displayed in the video playback interface is converted from the local perspective of the first camera to a 360° panoramic perspective to obtain the panoramic orientation angle of the target object. Based on the panoramic orientation angle, the optimal target camera that places the target object at a preset position in the video frame is determined, so that the video playback interface can switch to the target video frame corresponding to the target camera.

[0069] In this embodiment, the target tracking information carries the target position information of the target object within the currently displayed frame of the video playback interface. At this time, the camera identifier of the first camera corresponding to the currently displayed frame of the video playback interface, the corresponding horizontal field of view of the first camera, and the total number of cameras can also be obtained. Based on this information, a camera selection algorithm is used to complete the conversion between the three perspectives mentioned above, achieving accurate selection of the target camera.

[0070] Specifically, based on the horizontal field of view of the first camera and the camera identifier of the first camera, a preset first orientation angle of the first camera is determined. Then, based on the target position information and the preset first orientation angle of the first camera, the image displayed in the video playback interface is converted from the local view of the first camera to a 360° panoramic view, obtaining the panoramic orientation angle of the target object in the 360° panoramic space. Finally, based on the panoramic orientation angle and the horizontal field of view, the target camera corresponding to the optimal viewing angle to which the camera should be switched is determined.

[0071] The process of obtaining the panoramic orientation angle of the target object in the 360° panoramic space includes: determining the angular offset of the target object relative to the center of the image captured by the first camera based on the target position information, and then determining the panoramic orientation angle of the target object based on the angular offset and the preset first orientation angle of the first camera.

[0072] Specifically, determining the target camera to which the optimal viewing angle should be switched based on the panoramic orientation angle and the horizontal field of view includes: since the panoramic orientation angle is the orientation angle of the target object in the panoramic view, it can be determined by setting the panoramic orientation angle θ. global With horizontal field of view θ fov The ratio between them is used to determine the optimal target camera C in the panoramic space. opt :C opt =[θ global / θ fov ].

[0073] In practical applications, if the target camera C opt If the total number of cameras is N, then select the target camera C. opt =0.

[0074] In the above embodiments, based on the target location information, by switching between the local view of the first camera → the panoramic view → the optimal view of the target camera, the target camera that can make the target object appear in the preset position on the screen can be accurately determined, thereby improving the accuracy of video frame switching in the subsequent video playback interface.

[0075] In other embodiments of this application, determining the panoramic orientation angle of the target object based on the target location information and a preset first orientation angle of the first camera includes: Based on the target location information, determine the positional offset ratio of the target object relative to the center of the image captured by the first camera; based on the positional offset ratio, determine the angular offset of the target object relative to the center of the image captured by the first camera; based on the angular offset and the preset first direction angle of the first camera, determine the panoramic direction angle of the target object.

[0076] In this embodiment, the target object is subsequently displayed in the center of the video playback screen. The position offset ratio of this application is the horizontal offset ratio, which is used to describe the proportion by which the target object is offset from the center of the image captured by the first camera in the horizontal direction.

[0077] In this embodiment, firstly, the target location information is mapped onto the image captured by the first camera to determine the positional offset ratio of the target object relative to the center of the image captured by the first camera. For example, let the horizontal coordinate of the target location information be x. c Let w be the width of the image, and w / 2 be the x-coordinate of the image center. Then, the expression for calculating the position offset ratio r is: r = (x... c -w / 2) / 2.

[0078] Then based on the preset horizontal field of view θ of the first camera fov The position offset ratio is converted into an angular offset. The angular offset Δθ is actually the ratio of the position offset ratio r to the preset horizontal field of view θ of the first camera. fov The product of Δθ = r × θ fov For example, if the target location information indicates that the target object is located in the left 1 / 4 of the current frame, and the horizontal field of view θ fov If the angle is 10°, then the position offset ratio r = 0.5, angular offset Δθ= 5°.

[0079] Since the angular offset is the offset of the target object relative to the center of the image captured by the first camera, and the target object is currently in panoramic space, it can be considered that the panoramic orientation angle of the target object is equivalent to an offset of a certain angle based on the preset first orientation angle of the first camera. This offset angle is the angular offset calculated above. Furthermore, based on the angular offset and the preset first orientation angle C of the first camera... t θ fov The obtained panoramic orientation angle is the sum of the first orientation angle and the angle offset θ. global =(C t θ fov +Δθ)mod360°, mod360° means taking the remainder of 360°, converting any angle to the range of 0° to 360°.

[0080] In the above embodiments, by mapping the target position information onto the image captured by the first camera, the position offset ratio of the target object relative to the center of the image captured by the first camera is determined, and then the angular offset of the target object relative to the center of the image captured by the first camera is determined. This enables the panoramic direction angle of the target object to be accurately determined based on the angular offset and the preset first direction angle of the first camera, thereby improving the accuracy of the target camera determination.

[0081] In other embodiments of this application, in response to an instruction operation regarding a target object within a target area displayed on the video playback interface, an object detection request is sent to the cloud, including: In response to an instruction operation targeting a target object within the target area displayed on the video playback interface, the system determines the timestamp information triggered by the instruction operation, the instruction location information corresponding to the instruction operation, and the camera identifier of the second camera; based on the instruction location information, timestamp information, and camera identifier of the second camera, it generates an object detection request corresponding to the instruction operation; and sends the object detection request to the cloud.

[0082] The second camera is the camera corresponding to the view of the instruction. The instruction location information refers to the two-dimensional coordinates of the target object within the view corresponding to the instruction.

[0083] In this embodiment of the application, the user's instruction operation on the target object in the target area displayed on the video playback interface has time attributes and direction attributes. Therefore, when the user makes an instruction operation, the client responds to the instruction operation, determines the timestamp information t that triggers the instruction operation and the instruction direction vector corresponding to the instruction operation, and maps the instruction direction vector to the screen corresponding to the timestamp information to obtain the instruction position information (u, v) of the target object. The timestamp information is used to trigger the cloud to track the target object.

[0084] Simultaneously, the client can also obtain the camera identifier of the second camera corresponding to the screen displayed by the instruction operation. Based on the instruction location information (u, v), timestamp information, and the camera identifier of the second camera, it generates an object detection request corresponding to the instruction operation and sends the object detection request to the cloud.

[0085] Based on timestamp information and camera identifier, the cloud determines the first video frame corresponding to the instruction operation. This first video frame is the video frame that the cloud needs to detect the target object. Then, based on the instruction position information (u, v), the cloud locates the target object in the first video frame and continuously tracks the target object in each subsequent cycle to obtain the target tracking information of the target object in each cycle.

[0086] In the embodiments of this application, in response to an instruction operation on a target object in a target area within the video playback interface, what is actually obtained is the instruction direction vector of the instruction operation in the three-dimensional space where the user is located. It is also necessary to map the instruction direction vector onto the screen corresponding to the instruction operation in order to determine the instruction position information triggered by the instruction operation. The specific steps include: combining the instruction direction vector with the preset second direction angle of the second camera corresponding to the screen indicated by the instruction operation to determine the instruction position information of the target object indicated by the instruction operation, that is, mapping the instruction direction vector from the three-dimensional space where the user is located to the screen of the video playback interface to obtain the two-dimensional coordinate information (u, v) of the target object in the screen displayed on the video playback interface corresponding to the instruction operation. This two-dimensional coordinate information is the instruction position information.

[0087] Specifically, the determination of the indication position information triggered by the indication operation, by combining the indication direction vector with the preset second direction angle of the second camera corresponding to the image indicated by the indication operation, includes: determining the offset angle of the indication direction vector relative to the center of the image corresponding to the indication operation based on the indication direction vector and the preset second direction angle of the second camera, and determining the indication position information of the indication operation based on the offset angle.

[0088] Furthermore, when determining the indication position information corresponding to the indication operation, a preset resolution of the video playback interface can be introduced. This preset resolution can be configured independently and is used to determine whether the indication operation exceeds the field of view of the second camera. That is, the indication position information corresponding to the indication operation can be determined based on the indication direction vector, the preset second direction angle of the second camera, and the preset resolution of the video playback interface.

[0089] In practical applications, the indicated location information can also be determined by the user through touch or by the user directly selecting specific indicated location information through other means.

[0090] In the above embodiments, an object detection request is generated and sent to the cloud by using the timestamp information triggered by the instruction operation, the instruction location information corresponding to the instruction operation, and the camera identifier of the second camera. When the cloud receives the object detection request, it can accurately determine the video frame to be detected based on the timestamp information and camera identifier carried in the object detection request, and accurately detect the target object from the video frame based on the instruction location information.

[0091] In this embodiment of the application, determining the indication location information corresponding to the indication operation includes: Obtain the indication direction vector corresponding to the indication operation; determine the yaw angle of the indication direction vector relative to the preset horizontal plane; determine the offset angle of the indication direction vector relative to the center of the image corresponding to the indication operation based on the yaw angle and the preset second direction angle of the second camera; if the offset angle indicates that the indication operation does not exceed the field of view of the second camera, then determine the normalized offset ratio corresponding to the indication direction vector based on the offset angle; determine the indication position information of the indication operation based on the normalized offset ratio.

[0092] Here, the indicating direction vector refers to the direction of the operation within the user's 3D space. The yaw angle is the angle between the indicating direction vector and the horizontal plane in the user's 3D space, using the horizontal plane as a reference. The offset angle is the angle between the indicating direction vector and the center of the image captured by the second camera, using the center of the image captured by the second camera as a reference.

[0093] In this embodiment, the client responds to the instruction operation on the target object in the target area displayed on the video playback interface, determines the instruction direction vector in the three-dimensional space where the user is located, and determines the angle between the instruction direction vector and the preset horizontal plane. This angle is the yaw angle. Specifically, the yaw angle is yaw = arctan2(dx, dz) × 180 / π, where dx refers to the x-direction offset of the instruction direction vector in the three-dimensional space relative to the preset horizontal plane, and dz refers to the z-direction offset of the instruction direction vector in the three-dimensional space relative to the preset horizontal plane.

[0094] Next, the yaw angle of the indicator direction vector in three-dimensional space relative to the preset horizontal plane is mapped to the screen corresponding to the indicator operation. Since the screen corresponding to the indicator operation is captured by the second camera, it is necessary to first obtain the preset second direction angle θcenter=C of the second camera. t ×θ fov , where θ fov C is the horizontal field of view of the second camera. t This is the camera identifier for the second camera, and the camera identifier is determined according to the order in which the cameras are arranged.

[0095] Furthermore, by combining the yaw angle with the preset second orientation angle of the second camera, the indication direction vector is mapped to the screen corresponding to the indication operation, obtaining the offset angle Δθ of the indication direction vector relative to the center of the screen corresponding to the indication operation. The offset angle Δθ of the indication direction vector relative to the center of the screen corresponding to the indication operation is the yaw angle of the indication direction vector relative to the preset horizontal plane minus the inherent preset second orientation angle θ of the second camera. center The difference, Δθ=yaw θ center .

[0096] Finally, based on the offset angle of the indicator direction vector relative to the center of the screen corresponding to the indicator operation, the indicator position information of the indicator direction vector relative to the center of the screen corresponding to the indicator operation can be determined.

[0097] Before determining the indicated location information, it is also necessary to determine whether the indicated operation exceeds the field of view of the second camera. The specific determination process can be determined by the offset angle of the indicated direction vector relative to the center of the image corresponding to the indicated operation, that is, whether the absolute value of the offset angle |Δθ| is greater than the horizontal field of view θ of the second camera. fov Half of, where the horizontal field of view θ fov This refers to the angle between the two outermost imaging rays of a camera lens on a horizontal plane; if |Δθ|>θ fov If the value is 2, it means the instruction operation is outside the current view range and returns invalid; otherwise, the step of determining the instruction position information of the target object indicated by the instruction operation based on the offset angle continues.

[0098] In this embodiment of the application, the method for determining the indication position information of the indication operation based on the offset angle includes: normalizing the offset angle to the horizontal ratio of the screen to obtain the normalized offset angle, and then determining the indication position information of the target object indicated by the indication operation based on the normalized offset angle.

[0099] Furthermore, the horizontal field of view of the second camera is [-θ fov / 2,+θ fov / 2], compressing the horizontal field of view range by θ fov / 2 times to the standard scaling range [-1, +1], then at this time, the normalized offset angle r x Also based on the offset angle Δθ and half of the horizontal field of view θ corresponding to the second camera. fov / 2 yields r x =Δθ / (θ fov / 2).

[0100] Then, based on the normalized offset angle, the indicator direction vector is mapped to the indicator position information on the screen corresponding to the indicator operation. During this process, the preset resolution of the video playback interface is introduced. Combining the preset resolution with the normalized offset angle, the indicator position information of the target object on the screen corresponding to the indicator operation is determined. More specifically, the preset resolution includes the width (number of horizontal pixels) w and height (number of vertical pixels) h of the video playback interface. Based on the number of horizontal pixels of the video playback interface and the normalized offset angle, the indicator direction vector is mapped to a horizontal coordinate within the screen. Then, based on the number of vertical pixels of the video playback interface, the indicator direction vector is mapped to a vertical coordinate within the screen. Combining the horizontal and vertical coordinates of the indicator direction vector yields the pixel coordinates of the target object within the corresponding screen, i.e., the indicator position information.

[0101] The expression for calculating the indicated location information (u, v) is: u = w × [(r x +1)] / 2,v=h / 2。

[0102] It needs to be explained that, since the image pixels are [0, w], r x ∈[ 1, 1], therefore, the range needs to be expanded from [ To convert the coordinates from [1, 1] to [0, 2], the coordinates must be shifted to the pixel coordinates on the screen. Therefore, by using r... x +1∈[0,2], then [(r x +1)] / 2 is compressed to a [0,1] ratio, and finally multiplied by the width of the preset resolution to map the direction vector to the horizontal coordinate information in the screen. The vertical coordinate information in the screen is half the height of the video playback interface.

[0103] In the above embodiments, by combining the yaw angle of the indication direction vector relative to the preset horizontal plane with the preset second direction angle of the second camera, the indication direction vector can be accurately mapped to the center of the screen corresponding to the indication operation, and the offset angle of the indication direction vector relative to the center of the screen corresponding to the indication operation can be obtained. Then, when the indication operation does not exceed the field of view of the second camera, the indication position information of the target object in the screen corresponding to the indication operation can be accurately determined based on the offset angle.

[0104] Figure 4 A flowchart illustrating a target object display method provided in this application embodiment, taking the method applied to the cloud as an example, the target object display method provided in this application includes the following steps: S401, Get the object detection request sent by the client.

[0105] Among them, the object detection request is generated based on the instruction operation; the instruction operation is the instruction operation for the target object in the target area within the screen displayed on the video playback interface; the video playback interface displays the screen captured by any one of the preset multiple cameras, which are arranged around the target area and capture the target area.

[0106] In this embodiment, when watching a video playback interface, if a user wants to focus on a target object within a target area of ​​the screen, the user can use gestures or head rotation to indicate the target object in that area. The client then responds to this indication, generates an object detection request, and sends it to the cloud. The cloud receives the object detection request from the client and performs target tracking based on it.

[0107] In this embodiment, the object detection request includes at least a timestamp information of the indication operation triggered, indication location information corresponding to the indication operation, and a camera identifier of the second camera, wherein the second camera is the camera corresponding to the image indicated by the indication operation. The cloud subsequently obtains the timestamp information of the indication operation triggered, the indication location information corresponding to the indication operation, and the camera identifier of the second camera to perform target tracking of the target object.

[0108] S302 performs target tracking on the target object and periodically sends target tracking information to the client.

[0109] Specifically, the target tracking information instructs the client to identify the target camera from a set of preset cameras and displays the target video frames captured by that camera on the video playback interface. The target tracking information includes the target object's location information and its object identifier.

[0110] In this embodiment, the cloud determines the target object indicated by the instruction operation based on the object detection request, continuously tracks the target object, and periodically sends target tracking information to the client. In practical applications, the interval can be configured according to requirements, such as pushing the latest target tracking information of the target object to the client every 100ms. Then, based on the target tracking information sent by the cloud, the client determines the target camera from a set of preset cameras and displays the target video frame captured by the target camera on the video playback interface, with the target object in the target video frame positioned at a preset location.

[0111] Specifically, the cloud-based system analyzes the frame indicated by the instruction based on the object detection request using a target detection algorithm to identify the target object within the frame. More specifically, the object detection request includes at least the timestamp information triggered by the instruction, the corresponding location information, and the camera identifier of the second camera. The cloud uses these information to determine the target object and its corresponding location information. Then, a unique object identifier is assigned to the target object, and a multi-target tracking algorithm is activated. This algorithm continuously identifies and updates the target object's location information in subsequent video frames, ensuring that the object identifier is not lost. Within each cycle, the cloud generates tracking information for the target object based on its location and identifier.

[0112] Among them, the object detection algorithm includes, but is not limited to, calling a pre-trained object detection model. The object detection model can be a variety of detection models such as YOLO series and Faster R-CNN. The multi-object tracking algorithm includes, but is not limited to, multi-object tracking algorithms based on SORT or Kalman filtering.

[0113] In the above scheme, the client responds to an instruction on a target object within any frame of the video playback interface by sending an object detection request to the cloud. The video playback interface displays footage captured by any one of a set of pre-set cameras, which are positioned around the target area and capture images of it. The cloud tracks the target object based on the object detection request and periodically sends target tracking information to the client. The client then identifies the target camera from the pre-set cameras based on the target tracking information sent by the cloud. The target video frame captured by the target camera is displayed on the video playback interface, where the target object is positioned at a pre-set location within the video frame. Throughout this process, the client instructs the cloud to continuously track the target object and periodically send target tracking information to the client. This allows the client to automatically switch to the target video frame corresponding to the target camera at the appropriate intervals, eliminating the need for frequent manual switching and significantly reducing the operational burden. This not only keeps the target object displayed at the pre-set position on the video playback interface but also improves the smoothness of video playback.

[0114] In other embodiments of this application, the object detection request carries indication location information corresponding to the indication operation, timestamp information of the indication operation triggering, and camera identifier of the second camera; the second camera is the camera corresponding to the image indicated by the indication operation; target tracking of the target object includes: Within the first cycle, the first video frame of the target object captured by the second camera is determined based on the timestamp information and the camera identifier; the first position information of the target object in the first video frame is determined based on the indicated position information; and the target tracking information of the target object is generated based on the first position information and the object identifier of the target object.

[0115] The first cycle is the time period corresponding to the user's instruction.

[0116] In this embodiment, during the first cycle, the client maps the instruction operation to an instruction direction vector adapted to the currently displayed screen of the video playback interface. Based on the instruction direction vector, the client determines the instruction position information of the target object and uses this instruction position information as input information for the target detection algorithm in the cloud. Simultaneously, the client also sends the timestamp information triggered by the instruction operation and the camera identifier of the second camera corresponding to the screen indicated by the instruction operation to the cloud. The object detection request obtained by the cloud carries the instruction position information, timestamp information, and camera identifier of the second camera. Then, the cloud uses the target detection algorithm to track the target object based on the instruction position information, timestamp information, and camera identifier of the second camera, obtaining the target tracking information of the target object within this cycle, and sends the target tracking information to the client. The target detection algorithm can be a YOLO series detection algorithm, or it can be a Faster R-CNN, ATSS (Adaptive Training Sample Selection), etc., and is not limited here.

[0117] In this embodiment of the application, after obtaining the indication location information of the target object indicated by the indication operation, the timestamp information of the indication operation trigger, and the camera identifier of the second camera in the cloud, it is first necessary to determine the first video frame of the target object captured by the second camera based on the timestamp information and the camera identifier, that is, to determine the first video frame that matches both the obtained timestamp information and the camera identifier.

[0118] It should be noted that although the indicated location information is the location information of the target object indicated by the user within the screen corresponding to the indicated operation, there may be a small difference between the user's indicated operation and the actual location of the target object. Therefore, the indicated location information cannot be used as the accurate actual location information of the target object in the first video frame. Therefore, it is necessary to optimize the indicated location information to obtain the first location information of the target object in the first video frame. This first location information is used as the actual location information of the target object in the first video frame. Then, the first location information is combined with the object identifier of the target object to generate the target tracking information of the target object.

[0119] In this embodiment of the application, the optimization of the indicated location information can be achieved by: performing target detection on all objects in the first video frame using a target detection algorithm, and then determining the first location information that matches the indicated location information based on the detection results; in addition, the indicated location information can be pushed to the client's location correction interface to obtain the user's adjustment amount for the indicated location information, and then the accurate first location information of the target object can be obtained based on the adjustment amount and the indicated location information.

[0120] In the above embodiments, the indication location information corresponding to the indication operation may differ slightly from the actual location of the target object. This embodiment optimizes the indication location information to obtain the first location information of the target object. This first location information is more consistent with the actual location information of the target object in the first video frame. Therefore, the target tracking information generated based on the first location information and the object identifier is more accurate.

[0121] In other embodiments of this application, determining the first location information of the target object in the first video frame based on the indicated location information includes: Determine the first detection bounding boxes of multiple objects in the first video frame and the confidence of each first detection bounding box; determine the matching degree between the indication position information and each first detection bounding box; determine the second detection bounding box corresponding to the target object from the multiple first detection bounding boxes based on the matching degree and confidence degree corresponding to each first detection bounding box; determine the first position information of the target object in the first video frame based on the second detection bounding box.

[0122] In this context, the detection box refers to the rectangular frame that the object detection algorithm uses to define the location and size of the target object in the image. The confidence score of the detection box is a probability value between 0 and 1, used to determine whether an object exists within the detection box or the probability of the object belonging to a category. The final output confidence score = probability of object presence x category probability. The closer the confidence score is to 1, the more likely that an object exists within the detection box; the closer the confidence score is to 0, the more likely it is a false detection or background interference.

[0123] In this embodiment, the target detection algorithm supports target detection on the screen indicated by the instruction operation, and outputs the first detection box of each object in the screen and the confidence score corresponding to the first detection box. It should be explained that the object includes not only the target object, but also other objects in the screen corresponding to the instruction operation. For example, when the target object is a sphere, other objects can be all athletes, referees, etc. in the screen.

[0124] The object detection algorithm outputs first detection boxes for multiple objects, each corresponding to an accurate location. However, it's impossible to determine which detection box is the second detection box for the template object. Therefore, it's necessary to determine the second detection box for the target object from the first detection boxes of the multiple objects. At this point, the indicator location information corresponding to the previously obtained indicator operation can be used to match the indicator location information with each first detection box, obtaining the matching degree between the indicator location information and each first detection box, thus determining the most likely target object.

[0125] Then, based on the matching degree and confidence degree corresponding to each first detection box, the second detection box that best matches the indicated position information is determined, and the center position information of the second detection box is determined as the first position information of the target object in the first video frame.

[0126] In this embodiment of the application, the method for matching the indication location information with each first detection box includes: determining the Euclidean distance between the indication location information and the center location information of each first detection box, and determining the matching degree between the indication location information and the corresponding first detection box based on the Euclidean distance; or, determining the intersection-union ratio (IUU) between the indication location information and the center location information of each first detection box, and determining the matching degree between the indication location information and the corresponding first detection box based on the IUU. The matching algorithm can be selected according to actual needs and is not limited here.

[0127] Taking the determination of Euclidean distance as the matching degree between the indicated location information and the corresponding first detection box as an example, the steps to determine the target object and the first location information of the target object are as follows: 1. For each first detection box b i The object detection algorithm is used to output the first detection boxes of multiple objects in the first video frame and the confidence score of each first detection box. The output list of first detection boxes is B={b1, b2, ..., b n}, b i =(x i y i w i h i ), where x i y i w i h i These represent the x-coordinate, y-coordinate, width, and height of the reference point of the first detection box, respectively. The center position information of the center point of each first detection box is (x...). i +w i / 2,y i +h i / 2), where n is the identifier of the first detection box.

[0128] Calculate the Euclidean distance d between the center position information of the center point of each first detection box and the indication position information (u, v). i :

[0129] 2. Find the first detection box b with the smallest Euclidean distance. min And check whether its confidence level is greater than the threshold (e.g., 0.7); 3. If the conditions are met, the first detection box is identified as the second detection box corresponding to the target object, and the second detection box is marked as the target P selected by the user. ; 4. If no first checkbox meets the conditions, return "Selection failed".

[0130] In this embodiment, the method for determining the second detection box that best matches the indicated location information based on the matching degree and confidence level of each first detection box is as follows: Each first detection box is sorted according to its matching degree, and the first detection box with the highest matching degree and a confidence level greater than a preset confidence threshold is determined as the second detection box. For example, if there are first detection box 1 (matching degree 0.9, confidence level 0.8), first detection box 2 (matching degree 0.8, confidence level 0.6), and first detection box 3 (matching degree 0.5, confidence level 0.6), then first detection box 1 is determined as the second detection box. Further, if at least two first detection boxes have the same and highest matching degree, then the first detection box with the highest confidence level is determined as the second detection box.

[0131] In this embodiment of the application, if this is the first time that the user has selected target P, (That is, within the first cycle), it is then checked whether the first detection box matches the user-selected target P. Match; if a match is successful, select target P for the matched user. Assign a unique tracking ID (such as ID) focus =1001); if no match is found, activate the "short-term loss recovery mechanism" (such as re-identification based on appearance features, which can be selected according to actual needs); mark the object as "requires continuous push".

[0132] In the above embodiments, the first detection boxes of multiple objects output by the target detection algorithm are accurate location information. Therefore, the first detection boxes can be matched with the indicated location information to determine the second detection box corresponding to the target object from the multiple first detection boxes. Then, the center location information of the second detection box is used as the first location information of the target object in the first video frame. This first location information not only matches the indicated location information, but is also the real location information determined by the target detection algorithm. Therefore, the positioning process of the target object in the first period is more accurate.

[0133] In other embodiments of this application, target tracking of the target object includes: Within the Nth cycle, determine the second video frame captured by the first camera; acquire the historical tracking status information of the target object; determine the second position information of the target object in the second video frame based on the historical tracking status information; and determine the target tracking information of the target object based on the second position information and the object identifier of the target object.

[0134] Here, the first camera is the camera corresponding to the current image displayed on the video playback interface; N is a positive integer not equal to 1. Historical tracking status information refers to the target object's position information, velocity information, object identifier, and number of consecutive lost frames (used for object identifier management) in the previous cycle or frame. Based on historical tracking status information, the position information in the future period can be predicted.

[0135] In this embodiment of the application, when the object is first identified and marked as the target object P Subsequently, through cross-frame continuous matching and object identification maintenance mechanism, it is ensured that the target object is stably tracked in subsequent video sequences. Even if there is brief occlusion, viewpoint switching or detection fluctuation, its identity can still be maintained, providing a stable position data stream for the client to achieve continuous preset position display.

[0136] In the subsequent Nth cycle, in order to track the target object, the cloud will first determine the current screen displayed on the video playback interface, that is, determine the second video frame captured by the camera corresponding to the current screen displayed on the video playback interface, so as to locate the target object in the second video frame.

[0137] Since the user will not make any instructions during the current period, the target detection algorithm cannot use the indicated location information to assist in localization as it could in the first period. Therefore, the target object's historical tracking state information can be used to determine the target object's second location information in the second video frame. Finally, based on the second location information and the target object's object identifier, the target tracking information for that period is determined.

[0138] In this embodiment, determining the second position information of the target object in the second video frame based on historical tracking status information can be achieved by directly predicting the second position information of the target object in the second video frame within the nth period based on historical tracking status information, such as historical speed information and historical position information, while moving forward at historical speed information. To more accurately determine the second position information of the target object in the second video frame, target detection can also be performed on the second video frame using a target detection algorithm, and the target detection results can be combined with historical tracking status information to jointly determine the second position information of the target object in the second video frame.

[0139] In the above embodiments, by using historical tracking status information, the target tracking information of the target object in subsequent periods can be accurately predicted, so that the user does not need to make repeated instruction operations, and the video frames displayed on the video playback interface can be accurately switched, thus improving the intelligent viewing experience.

[0140] In other embodiments of this application, determining the second location information of the target object in the second video frame based on historical tracking status information includes: Based on historical tracking status information, predict the third location information of the target object in the second video frame; determine the third detection boxes of multiple objects in the second video frame, and the confidence level of each third detection box; based on the matching degree between the third location information and each third detection box, and the confidence level corresponding to each third detection box, determine the fourth detection box of the target object from the multiple third detection boxes; based on the fourth detection box, determine the second location information of the target object in the second video frame.

[0141] Among them, historical tracking status information is used to predict the third position information of the target object in the currently displayed frame of the video playback interface, so as to realize the continuation of object identification.

[0142] In this embodiment, since the motion trajectory of the target object is practically unpredictable, the third position information of the target object in the second video frame predicted based on historical tracking status information is still not accurate enough. In this case, the cloud can also use a target detection algorithm to combine the third position information to determine the accurate second position information of the target object.

[0143] Specifically, by using an object detection algorithm to determine the third detection boxes of multiple objects in the second video frame, and the confidence score of each third detection box, a set of third detection boxes B can be obtained. t ={b1, b2, ..., b n}, where each third detection box b i =(x i y i w i h i ), x i y i w i h i The reference point x-coordinate, reference point y-coordinate, width, and height of the third detection box representing the detected object are used to characterize the position and size of the object; the third detection box, and the confidence level of each third detection box, serve as auxiliary reference information for the third position information.

[0144] Since the target detection algorithm obtains the third detection box of all objects in the second video frame, it is also necessary to determine the fourth detection box of the target object from the third detection boxes of all objects. At this time, the third position information of the target object in the second video frame can be predicted based on the historical tracking state information. The third position information can be matched with the third detection box, and the fourth detection box that best matches the third position information can be determined from multiple third detection boxes according to the matching result. Then, the center position information of the fourth detection box is determined as the second position information of the target object in the second video frame.

[0145] Ultimately, the cloud receives a list of tracking information with object identifiers, T t ={(b1, ID1), (b2, ID2), ...}, where each third detection box b i Associate with a stable object identifier ID i Specifically, the target object P is output. Corresponding target tracking information (ID) focus b focus (t), which can be pushed to the client later; in addition, the cloud can also push the current tracking status information, such as whether the ID is lost or newly activated, to the client.

[0146] In this embodiment, the method for determining the fourth detection box of the target object from multiple third detection boxes includes: determining the fourth detection box of the target object from multiple third detection boxes based on the matching degree between the third position information and each third detection box, and the confidence degree corresponding to each third detection box. Specifically, each third detection box is sorted in descending order of matching degree, and the third detection box with the highest matching degree and a confidence degree greater than a preset confidence threshold is determined as the fourth detection box corresponding to the target object. For example, if there are third detection boxes 4 (third matching degree 0.9, confidence degree 0.8), first detection box 5 (third matching degree 0.8, confidence degree 0.6), and third detection box 6 (third matching degree 0.5, confidence degree 0.6), then third detection box 4 is determined as the fourth detection box. Further, if there are at least two third detection boxes with the same and highest matching degree, then the third detection box with the highest confidence degree is determined as the fourth detection box.

[0147] In the embodiments of this application, the method for predicting the third position information of the target object in the second video frame based on historical tracking status information includes, but is not limited to: processing the historical tracking status information using the Kalman filter method to predict the third position information of the target object in the second video frame, or using the polynomial curve fitting method in combination with the historical tracking status information to predict the third position information of the target object in the second video frame.

[0148] In this embodiment, the matching degree between the third location information and each third detection box can be determined by the Euclidean distance between the third location information and each third detection box, or by the intersection-union ratio between the third location information and each third detection box. The specific matching degree determination method can be selected according to actual needs.

[0149] In the embodiments of this application, the prediction of the second position information can also be completed by combining algorithms such as SORT (Simple Online and Realtime Tracking) or YOLO-Track. Among them, SORT (Simple Online and Realtime Tracking) is a lightweight online multi-target tracking algorithm. Its core is detection + Kalman filtering + Hungarian matching, which can realize multi-target tracking at a higher speed and in real time.

[0150] Taking the SORT algorithm for target tracking as an example, the process of determining the second position information of the target object in the second video frame is as follows: 1. Prediction Phase: For each existing target object k that needs to be tracked, Kalman filtering is used to predict its second position information in the second video frame. Specifically, the state transition matrix, process noise covariance matrix, and covariance matrix of the target object are obtained, and the tracking state information X of the target object in the second video frame is determined based on the historical tracking state information, state transition matrix, process noise covariance matrix, and covariance matrix of the target object in the previous frame. t|k And based on the tracking status information X t|k Determine the second location information of the target object: X t|k =Fx t 1|k P t|k =FP t 1|k F T +Q Where: x t 1|k : Historical tracking state information, such as the state vector (position, velocity) of the previous frame; F: State transition matrix; F T Q: Transpose of the state transition matrix; X: Process noise covariance matrix; t|k Current tracking status information; P t 1|k : Covariance matrix of the previous frame; P t|k : The predicted covariance matrix.

[0151] 2. Matching Phase (Association) Using the Hungarian algorithm, the predicted third location information b is calculated based on the IoU (Intersection over Union). i The third detection box b for each object obtained by the object detection algorithm j Perform matching to obtain the matching degree (cost) between the third location information and each third detection box. ij : cost ij =1 IoU(b i b j ) If the matching degree cost between the third location information and each third detection box is... ij If the value exceeds the preset threshold, a successful match is considered. The matched third detection box is then designated as the fourth detection box of the target object, and the second position information of the target object is determined based on the center position information of the fourth detection box. Unmatched third detection boxes are considered new targets, and unmatched target objects are marked as temporarily lost.

[0152] In the above embodiments, when the user does not trigger an instruction operation on the currently displayed screen of the video playback interface, the second position information predicted by the historical tracking status information is matched with the third detection box determined by the target detection algorithm to determine the second position information of the target object in the second video frame. This second position information not only matches the historical tracking status information, but is also the real position information determined by the target detection algorithm. Therefore, the positioning of the target object in the Nth period is more accurate.

[0153] To more clearly describe the target object display method of this application, this application uses a user watching a live basketball game on a basketball court as an example to describe a target object display method. In this method, 36 cameras are arranged around the 360° panoramic space of the basketball court, including camera 0, camera 1, ..., camera 35 arranged in clockwise order. Each camera captures an image of the basketball court, and each camera has a corresponding horizontal field of view θ. fov The angle is 10°, meaning that the directional angle corresponding to camera 0 is 0°, the directional angle corresponding to camera 1 is 10°, ..., and the directional angle corresponding to camera 35 is 350°.

[0154] To achieve the display of the target object, the specific interaction process between the user, the client, and the cloud is as follows: S1, User → Client: Users tap on athlete number 7 in the video playback interface using a gesture device (such as Leap Motion); the Leap Motion device captures the user's gesture in real time and sends the timestamp of the user's tap and the direction vector of the user's tap to the client.

[0155] S2, Client → Cloud: The client combines the orientation angle of camera 2 corresponding to the currently displayed image on the video playback interface, and uses a gesture-to-image coordinate mapping algorithm to map the gesture's direction vector in three-dimensional space to the current image's position information (u, v).

[0156] After obtaining the indication location information (u, v) on the client, the system generates an object detection request corresponding to the indication operation based on the timestamp information t of the user's click action, the indication location information (u, v) corresponding to the indication operation, and the camera identifier of camera 2, and sends the object detection request to the cloud.

[0157] Specifically, the algorithm for mapping gestures to screen coordinates includes: (1) Input parameters: The direction vector (unit vector) output by Leap Motion: d =(dx, dy, dz); C t ∈{0, 1, ..., 35}: The identifier C of the currently playing camera. t (i.e., camera is labeled 2); C t θ fov : The orientation angle of each camera, where θ fov θ is the horizontal field of view of each camera. fov =10 w=1920, h=1080: Width and height of the video frame resolution (configurable).

[0158] (2) Algorithm implementation logic: Calculate the yaw angle of the direction vector in the horizontal plane: yaw = arctan2(dx, dz) × 180 / π; the orientation angle of the center of the current camera 2 is: θ center =C t ×θ fov ; Calculate the offset angle Δθ=yaw of the gesture relative to the center of the currently displayed screen. θ center If |Δθ|>θ fov / 2 indicates that the gesture is outside the current view range, and the return value is invalid; otherwise, the offset angle is normalized to obtain the normalized offset angle as r. x :r x =Δθ / (θfov / 2); Map the normalized offset angle to pixel coordinates (u, v): u = w × [(r x +1)] / 2,v=h / 2。

[0159] (3) Output results: Pixel coordinates (u, v): The position information of the target object in the current screen where the gesture is pointed, which is used for subsequent target matching.

[0160] S3, Cloud → Client: After receiving the object detection request, the cloud first determines the first video frame of the target object captured by camera 2 at the timestamp corresponding to the timestamp information. Then, it calls the YOLO series object detection model to analyze the first video frame, obtaining the detection boxes and their confidence scores for all people and key objects (such as a ball) in the image. Subsequently, it uses an object detection algorithm to calculate the center point and indicated position information of each detection box. u , v The Euclidean distance between the two individuals is used to select the person who is closest to them and has a confidence level higher than a threshold (e.g., 0.7) as the target. P and for the target object P Assign a unique tracking ID.

[0161] In subsequent video frames, a multi-target tracking algorithm based on SORT or Kalman filtering is activated to continuously identify and update the target's position, ensuring that the ID is not lost.

[0162] The target detection algorithms include: (1) Input parameters: (u, v): Location information sent by the client, camera identifier of camera 2 (used to confirm which camera's viewpoint), and the detection box list B={b1, b2, ..., b} output by YOLO. n}

[0163] (2) Processing logic: For each detection box b i Calculate the detection box b i The Euclidean distance between the center location information and the indicator location information (u, v): Find the detection box b with the smallest distance. min The detection box is then checked to see if its confidence level is greater than a threshold (e.g., 0.7). If the condition is met, the detection box is marked as P. If no matching detection box is found, return "Selection failed".

[0164] Among them, SORT-based multi-target tracking algorithms include: (1) Input parameters: YOLO detection results of the current screen displayed on the video playback interface, historical tracking status information of the previous frame, including the predicted position (Kalman filter state vector) of each tracked target, velocity estimation, unique tracking ID (e.g., ID=1001), and number of consecutive lost frames (for ID management).

[0165] (2) Using the SORT (Simple Online and Realtime Tracking) algorithm, combined with Kalman filtering and Hungarian matching, the multi-target tracking process is as follows: For each existing tracked target k, use Kalman filtering to predict its position information in the current video frame t: X t|k =Fx t 1|k P t|k =FP t 1|k F T +Q Where: x t 1|k : Historical tracking state information, such as the state vector (position, velocity) of the previous frame; F: State transition matrix; F T Q: Transpose of the state transition matrix; X: Process noise covariance matrix; t|k Current tracking status information; P t 1|k : Covariance matrix of the previous frame; P t|k : The predicted covariance matrix.

[0166] Then, using the Hungarian algorithm, the predicted location information b is calculated based on the IoU (Intersection over Union). i Multiple detection boxes in the current YOLO detection results bj Perform matching and obtain the matching degree cost. ij =1 IoU(b i , b j Based on the matching degree, the detection box corresponding to athlete number 7 is determined. Based on the center position information of the determined detection box, the center pixel coordinates (x, y) of athlete number 7 in the current camera view are determined. c , y c ).

[0167] (3) Output results: A list of tracking results with IDs: T t={(b1, ID1), (b2, ID2), ...}, where each detection box is associated with a stable ID of an object; the focus is on the persistent position of the target: special output P Corresponding (ID) focus b focus ,t), for subsequent push to the client; tracking status information: such as whether the ID is lost, whether it is newly activated, etc.

[0168] Finally, the cloud returns a response data packet to the client, containing: the unique tracking ID of athlete number 7. focus The center pixel coordinates (x, y) of athlete number 7 from the current camera view. c , y c Information such as current tracking status.

[0169] S4, Client: After receiving the target information, the client initiates a dynamic viewpoint centering algorithm to determine the optimal camera identifier C to switch to. opt It also smoothly switches the video stream to center the target person in the frame.

[0170] The dynamic viewpoint centering algorithm includes: (1) Input parameters: The center pixel coordinates (x, y) of athlete number 7 in the current camera view. c , y c The camera icon C corresponding to the currently playing video. t ∈{0, 1, ..., 35}, the orientation angle C of each camera t ×θ fov The width of the video playback interface is w (e.g., 1920 pixels), and the total number of cameras is N=36.

[0171] (2) Algorithm implementation logic: This algorithm achieves precise centering through a three-step transformation of "local coordinates → global orientation angle → optimal viewpoint": Calculate the horizontal position offset ratio r of the target relative to the current camera center, r = (x c -w / 2) / w, Where w / 2 is the x-coordinate of the center of the image.

[0172] Convert the offset ratio to an angular offset: Δθ = r × θ fov ; Calculate the panoramic orientation angle of athlete number 7 in a 360° panoramic space: θ global =(C t ×θ fov +Δθ)mod360 This angle represents the absolute orientation of the target in the annular space.

[0173] Finally, determine the optimal target camera identifier to switch to: C opt =[θ global / θ fov ]; If C opt If the value is ≥36, then take C. opt =0 (modulo 36 processing).

[0174] (3) Output result: Target camera identifier C opt ∈{0, 1, ..., 35}, the target camera identifier is used to indicate which camera's video stream the video playback interface should switch to.

[0175] Through the above process, a shift from "passive viewing" to "active focusing" has been achieved, resulting in automatic viewpoint tracking, smooth and natural switching, rich information presentation, and a significantly enhanced sense of immersion. It can be applied to 3D full-scale video environments, supporting natural gesture control and immersive dynamic playback in real-world scenarios such as sports events and concerts. This significantly expands the practical application boundaries of gesture interaction technology and enhances the comprehensive performance of multimodal systems in complex environments.

[0176] In other embodiments of this application, a full-fledged media player is also provided, taking gesture operation as an example, such as... Figure 5 As shown, the full-fidelity player includes at least a real-time display module, a gesture control module, and an intelligent analysis module. The full-fidelity player connects to the device. The real-time display module and the gesture control module are located on the client side, while the intelligent analysis module is located in the cloud. The real-time display module receives 3D full-fidelity video streams from multiple cameras. The gesture control module acquires gesture data sent by the gesture device for the target object displayed on the real-time display module, generates indication position information based on the gesture data, and inputs the indication position information to the cloud-based intelligent analysis module for analysis to obtain analysis data. The analysis data carries the target camera that makes the target object appear in a preset position. The analysis data is sent from the intelligent analysis module to the real-time display module on the client side, so that, in conjunction with the gesture control module, the switching of the target video frame captured from the perspective of the target camera can be completed on the video playback interface.

[0177] In other embodiments of this application, this application also employs a hybrid architecture of local lightweight interaction and cloud-based heavy-duty AI (Artificial Intelligence), such as... Figure 6 As shown, this architecture enables the target object to be displayed in the center of the video playback interface in a certain scene: First, on the client side: (1) The client's video playback interface displays the real-time image of the scene from a certain camera's perspective through a video player; (2) Users interact with the screen displayed on the client through gesture input.

[0178] (3) The client's gesture recognition module captures the input gesture and uses the gesture-to-screen coordinate mapping algorithm to perform coordinate mapping and processing to obtain the gesture coordinates of the target object.

[0179] (4) Based on the gesture coordinates, generate an object detection request and send the object detection request to the cloud to obtain the target tracking information of the target object.

[0180] Secondly, in the cloud: (1) Use YOLO and other technologies to detect targets, lock the target object pointed to by the gesture in the scene, and assign an object identifier to the target object.

[0181] (2) Use a multi-target tracking algorithm to continuously track the target object and update its target location information in real time.

[0182] (3) Periodically return the object identifier and target location information of the target object to the client.

[0183] Finally, on the client side: (1) Receive the object identifier and target location information of the target object.

[0184] (2) Based on the received target location information, the centering control algorithm module is used to dynamically adjust the target camera view required to be switched on the video playback interface, and the video player is used to switch the video playback interface to the target video frame captured by the target camera, in which the target object is in the center of the screen.

[0185] (3) The video player smoothly switches the video stream based on the periodically acquired target position information, so that the centered video stream is displayed seamlessly, providing a seamless viewing experience.

[0186] This application embodiment also provides a target object display system, the system including a client and a cloud, the system including: The client is used to send an object detection request to the cloud in response to an instruction operation on a target object in a target area displayed on the video playback interface; wherein, the video playback interface displays the image captured by any one of a set of preset cameras, and the preset cameras are arranged around the target area to capture the target area. In the cloud, it is used to track target objects and periodically send target tracking information to the client; The client is used to determine the target camera from multiple preset cameras based on target tracking information sent from the cloud; and to display the target video frames captured by the target camera on the video playback interface, wherein the target object in the target video frame is located at a preset position.

[0187] In this embodiment of the application, when watching the video playback interface, if the user wants to focus on a target object in the target area of ​​the screen, the user can use gestures or head rotation to indicate the target object in the target area of ​​the screen. At this time, the client can respond to the indication operation in a timely manner, obtain the indication location information corresponding to the indication operation, the camera identifier corresponding to the screen indicated by the indication operation, and the timestamp of the indication operation. Based on the indication location information corresponding to the indication operation, the camera identifier corresponding to the screen indicated by the indication operation, and the timestamp of the indication operation, an object detection request is generated and sent to the cloud.

[0188] Furthermore, based on the object detection request, and using the timestamp and camera identifier of the indicated operation in the object detection request, the cloud determines the first video frame corresponding to the indicated operation, and uses target detection algorithms such as YOLO series to determine the first detection boxes of multiple objects in the first video frame, as well as the confidence of each first detection box; based on the matching degree between the third location information and each first detection box, and the confidence of each first detection box, the cloud determines the second detection box of the target object from the multiple first detection boxes; based on the second detection box, the cloud determines the second location information of the target object in the second video frame; based on the second location information and the object identifier of the target object, the cloud generates the target tracking information of the target object in the first period, and assigns a special object identifier to the target object.

[0189] Within the Nth period, the cloud acquires historical tracking status information of the target object; based on the historical tracking status information, it predicts the third position information of the target object in the second video frame; and uses target detection algorithms such as YOLO series to determine the third detection boxes of multiple objects in the second video frame, as well as the confidence of each third detection box; based on the matching degree between the third position information and each third detection box, and the confidence of each third detection box, it determines the fourth detection box of the target object from the multiple third detection boxes; based on the fourth detection box, it determines the second position information of the target object in the second video frame; and based on the second position information and the object identifier of the target object, it determines the target tracking information of the target object.

[0190] The cloud periodically sends target tracking information to the client, and the client obtains the target object's target tracking information within each period. In practical applications, the interval between periods can be configured according to requirements, such as pushing the latest target tracking information of the target object to the client every 100ms.

[0191] Based on the target location information carried in the target tracking information and the preset first direction angle of the first camera, the panoramic direction angle of the target object is determined; based on the panoramic direction angle, the target camera is determined from a preset number of cameras, and the target video frame captured by the target camera is displayed on the video playback interface, so that the target object is displayed at a preset position on the video playback interface.

[0192] The more detailed interaction process between the client and the cloud is described in the above embodiments and will not be repeated here.

[0193] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0194] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0195] It should be noted that the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.

[0196] In addition, in the embodiments of the present invention, "first," "second," etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0197] The target object display method of this application embodiment can be implemented by a processor in an electronic device, such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU), or a field-programmable gate array (FPGA).

[0198] To implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device. Figure 7 This is a schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, the electronic device includes: The communication interface 701 enables information exchange with other devices, such as electronic devices. The processor 702 is connected to the communication interface 701 to enable information exchange with other devices and to execute the methods provided by one or more of the above-mentioned technical solutions when running a computer program. The computer program is stored in the memory 703.

[0199] Of course, in practical applications, the various components in an electronic device are coupled together through a bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 7 The general designated all buses as Bus System 704.

[0200] The memory 703 in this embodiment is used to store various types of data to support the operation of the computer device. Examples of such data include any computer program used to operate on the electronic device.

[0201] It is understood that memory can be volatile or non-volatile, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0202] The methods disclosed in the embodiments of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0203] Optionally, when the processor executes the program, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0204] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory storing a computer program, which can be executed by a processor of a computer device to complete the aforementioned method steps. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, computer devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0206] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0207] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0208] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0209] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0210] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 702 of an electronic device to complete the steps of the target object display method in this application embodiment.

[0211] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0212] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0213] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for displaying a target object, characterized in that, Applied to a client, the method includes: In response to an instruction operation on a target object within a target area displayed on a video playback interface, an object detection request is sent to the cloud; wherein, the video playback interface displays footage captured by any one of a plurality of preset cameras, the plurality of preset cameras being arranged around the target area and capturing the target area; the cloud performs target tracking on the target object based on the object detection request and periodically sends target tracking information to the client; Based on the target tracking information sent from the cloud, the target camera is determined from the preset plurality of cameras; The video playback interface displays target video frames captured by the target camera, wherein the target object in the target video frames is located at a preset position.

2. The method according to claim 1, characterized in that, The target tracking information includes the target position information of the target object in the image captured by the first camera; the first camera is the camera corresponding to the image currently displayed on the video playback interface; The step of determining the target camera from the preset plurality of cameras based on the target tracking information sent from the cloud includes: Based on the target location information and the preset first direction angle of the first camera, the panoramic direction angle of the target object is determined; the panoramic direction angle is the direction angle of the target object in the panoramic view; the panoramic view is generated by images captured by multiple preset cameras respectively; The target camera is determined from the preset plurality of cameras based on the panoramic orientation angle.

3. The method according to claim 1, characterized in that, The step of sending an object detection request to the cloud in response to an instruction operation on a target object within a target area displayed on the video playback interface includes: In response to an instruction operation on a target object within a target area displayed on the video playback interface, the timestamp information of the instruction operation, the instruction location information corresponding to the instruction operation, and the camera identifier of the second camera are determined; the second camera is the camera corresponding to the image indicated by the instruction operation. Based on the indicated location information, the timestamp information, and the camera identifier of the second camera, an object detection request corresponding to the indicated operation is generated; Send the object detection request to the cloud.

4. The method according to claim 3, characterized in that, Determining the indication location information corresponding to the indication operation includes: Obtain the indication direction vector corresponding to the indication operation; Determine the yaw angle of the indicated direction vector relative to a preset horizontal plane; Based on the yaw angle and the preset second direction angle of the second camera, determine the offset angle of the indication direction vector relative to the center of the screen corresponding to the indication operation; If the offset angle indicates that the instruction operation does not exceed the field of view of the second camera, then the instruction position information of the instruction operation is determined based on the offset angle.

5. A method for displaying a target object, characterized in that, Applied to the cloud, the method includes: Obtain an object detection request sent by the client, wherein the object detection request is generated according to an instruction operation; the instruction operation is an instruction operation for a target object in a target area within the screen displayed on the video playback interface; the video playback interface displays the screen captured by any one of a plurality of preset cameras, which are arranged around the target area and capture images of the target area; The target object is tracked, and target tracking information is periodically sent to the client. The target tracking information instructs the client to identify the target camera from the preset multiple cameras and display the target video frame captured by the target camera on the video playback interface.

6. The method according to claim 5, characterized in that, The object detection request carries the indication location information corresponding to the indication operation, the timestamp information of the indication operation, and the camera identifier of the second camera; The second camera is the camera corresponding to the image shown in the instruction operation; The target tracking of the target object includes: Within the first period, the first video frame of the target object captured by the second camera is determined based on the timestamp information and the camera identifier; Based on the indicated location information, determine the first location information of the target object in the first video frame; Based on the first location information and the object identifier of the target object, target tracking information of the target object is generated.

7. The method according to claim 5, characterized in that, The target tracking of the target object includes: Within the Nth period, determine the second video frame captured by the first camera; the first camera is the camera corresponding to the currently displayed image on the video playback interface; N is a positive integer not equal to 1; Obtain the historical tracking status information of the target object; Based on the historical tracking status information, determine the second location information of the target object in the second video frame; Based on the second location information and the object identifier of the target object, the target tracking information of the target object is determined.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

9. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.