Intelligent device and method for target tracking display

Abstract

The application discloses an intelligent device and a target tracking display method. When playing a free-viewpoint video stream and performing tracking display, the target to be tracked in each frame of the display is obtained after correction processing is performed when correction processing is needed, that is, each frame needs to be judged before each frame is displayed. If a frame needs to be corrected, the correction is performed, and then the tracking display is performed based on the target to be tracked after the correction. If a frame does not need to be corrected, the correction is not needed. In this way, the target to be tracked in each frame of the display is the target that the user wants to focus on, tracking errors are avoided, the accuracy of the tracking display is improved, and tracking errors and interference caused by problems such as occlusion and similar appearance are effectively eliminated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of free viewpoint, in particular to an intelligent device and a target tracking display method. BACKGROUND

[0002] Free viewpoint technology is a new video technology, which can be applied in live broadcast environment. Through accurate synchronization control of multiple cameras, pictures are continuously collected, which can ensure the synchronization and preservation of live broadcast highlights, so that users can watch the game with 360° free viewing angle. Moreover, when users watch free viewpoint video images, target tracking display can be realized.

[0003] Taking the application of free viewpoint technology in basketball game as an example, due to the problems such as similar appearance between players, fast moving speed of players and serious occlusion, tracking display may be wrong, which leads to the target displayed is not the target that users really want to focus on. SUMMARY

[0004] The embodiments of the present application provide an intelligent device and a target tracking display method, which can improve the accuracy of tracking display and eliminate tracking errors and interference caused by problems such as occlusion and similar appearance.

[0005] In a first aspect, the embodiments of the present application provide an intelligent device, which comprises:

[0006] a display configured to display pictures;

[0007] a processor configured to:

[0008] respond to a user's instruction of tracking display on a target to be tracked;

[0009] control the display to perform tracking display when displaying a free viewpoint video stream;

[0010] wherein, when performing the tracking display, the target to be tracked in each frame displayed is obtained after correction processing is performed when the target to be tracked needs to be corrected.

[0011] In a second aspect, the embodiments of the present application provide a target tracking display method, which comprises:

[0012] responding to an instruction of tracking display on a target to be tracked;

[0013] performing the following processing on the target to be tracked in each frame of a free viewpoint video stream: performing correction processing when the target to be tracked needs to be corrected.

[0014] The present application has the following advantages:

[0015] This invention provides a smart device and a target tracking and display method. When playing a free-viewpoint video stream and performing tracking display, the target to be tracked in each frame is obtained after correction processing when necessary. That is, each frame needs to be judged before displaying it. If a frame needs correction processing, it is corrected, and then tracking and display are performed based on the corrected target to be tracked. If a frame does not need correction processing, it is not required. In this way, the target to be tracked in each frame displayed is the target that the user wants to focus on, avoiding tracking errors, improving the accuracy of tracking display, and effectively eliminating tracking errors and interference caused by problems such as occlusion and similar appearance. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the operation scenario between the display device and the control device provided in an embodiment of the present invention;

[0017] Figure 2 This is a hardware configuration block diagram of the display device 200 provided in an embodiment of the present invention;

[0018] Figure 3 This is a hardware configuration block diagram of the control device 100 provided in an embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram of the software configuration in the display device 200 provided in an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram showing the icon control interface of an application in a display device 200 provided in an embodiment of the present invention;

[0021] Figure 6 This is a schematic diagram of the structure of a smart device provided in an embodiment of the present invention;

[0022] Figure 7 A flowchart of a specific embodiment provided in this invention;

[0023] Figure 8 This is a schematic diagram illustrating the tracking effect provided in an embodiment of the present invention;

[0024] Figure 9 This is a flowchart of a target tracking and display method provided in an embodiment of the present invention. Detailed Implementation

[0025] The embodiments of the intelligent device and the method for target tracking display provided by the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0026] Figure 1 For the schematic diagram of the operation scene between the display device and the control device according to one or more embodiments of the present application, as shown in Figure 1 The user can operate the display device 200 through the mobile terminal 300 and the control device 100. The control device 100 can be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired ways to control the display device 200. The user can input user instructions through the keys on the remote controller, voice input, control panel input, etc. to control the display device 200. In some embodiments, mobile terminals, tablets, computers, laptops, and other smart devices can also be used to control the display device 200.

[0027] In some embodiments, the mobile terminal 300 can install a software application with the display device 200, realize connection communication through a network communication protocol, and achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to realize synchronous display function. The display device 200 also communicates data with the server 400 through various communication methods. The display device 200 can be connected through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 can provide various content and interaction to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing broadcast receiving television functions, the display device 200 can also provide smart network television functions with computer support functions.

[0028] Figure 2 An exemplary configuration block diagram of the control device 100 according to an exemplary embodiment is shown. As shown in Figure 2As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a WiFi chip (wherein this invention, WIFI and WiFi have the same meaning and can be used interchangeably), a Bluetooth module, NFC, or a replacement module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, buttons, or a replacement module.

[0029] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown. For example... Figure 3 The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 can be located in different separate devices; that is, the tuner / demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0030] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. The user can input user commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0031] In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application program or an operating system and a user, which realizes the conversion between the internal form of information and the form that the user can accept. The commonly used form of the user interface is a graphic user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an icon, a window, a control, etc. interface element displayed in the display screen of an electronic device, wherein the control can include at least one of the following visual interface elements: an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a widget, etc.

[0032] Figure 4 For the software configuration diagram of the display device 200 according to one or more embodiments of the present application, as shown in Figure 4 , the system is divided into four layers from top to bottom, namely, an application layer (referred to as "application layer" for short), an application framework layer (referred to as "framework layer" for short), an Android runtime and system library layer (referred to as "system runtime library layer" for short), and a kernel layer. The kernel layer at least includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a WIFI driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power supply driver, etc.

[0033] Figure 5 For the icon control interface display diagram of the application program in the display device 200 according to one or more embodiments of the present application, as shown in Figure 5 , the application layer includes at least one application program, which can display a corresponding icon control in the display, such as: a live TV application program icon control, a video on demand application program icon control, a media center application program icon control, an application center icon control, a game application icon control, etc. The live TV application program can provide live TV through different signal sources. The video on demand application program can provide videos from different storage sources. Unlike the live TV application program, the video on demand provides video display from certain storage sources. The media center application program can provide various multimedia content playing application programs. The application center can provide storage of various application programs.

[0034] The embodiment of the present application provides a kind of intelligent equipment, as shown in Figure 6 , can include:

[0035] Display 601 is configured to display a picture;

[0036] The processor 602 is configured to:

[0037] in response to an instruction of a user to track display the target to be tracked;

[0038] The display 601 is controlled to track display when displaying the free-viewpoint video stream.

[0039] In the track display, the target to be tracked in each frame is obtained after correction when correction is needed.

[0040] When playing the free-viewpoint video stream and track displaying, the target to be tracked in each frame is obtained after correction when correction is needed, that is, each frame needs to be judged before being displayed, if a frame needs correction, the correction is performed, and then the track display is performed based on the target to be tracked after correction, if a frame does not need correction, no correction is needed; in this way, the target to be tracked in each frame is the target that the user wants to focus on, avoiding tracking errors and improving the accuracy of track display, thereby effectively eliminating tracking errors and interference caused by occlusion and similar appearance.

[0041] It should be emphasized that in the embodiment of the application, for each frame in the track display process, before being displayed, it is judged whether the target to be tracked in the current frame needs correction, if the target to be tracked in the current frame needs correction, it means that the target to be tracked in the current frame may be wrong and may not be the target that the user really wants to focus on, so correction is needed, and the display is performed based on the target to be tracked after correction, so as to eliminate tracking errors.

[0042] Of course, if it is judged that the target to be tracked in a frame does not need correction, it means that the target to be tracked in the frame is correct and accurate, so no correction is needed and the frame can be directly displayed.

[0043] Optionally, in the embodiment of the application, the intelligent device can be a display device or other device with display function.

[0044] Optionally, in order to be able to judge which frame the target to be tracked in needs correction, in the embodiment of the application, the processor is configured to:

[0045] determine the position information and the confidence of the target to be tracked in the current frame;

[0046] When it is judged that the position information, the confidence, and / or the frame sequence number of the current frame in the free-viewpoint video stream meets the preset correction condition, it is determined that the target to be tracked needs correction.

[0047] Specifically, in determining the position information and the confidence, the Siam tracking algorithm can be adopted, of course, other algorithms that can determine the position information and the confidence can also be adopted, which are not limited herein.

[0048] The confidence can be understood as a proportion of the to-be-tracked target existing in the current frame, the higher the proportion, the greater the possibility of the to-be-tracked target existing in the current frame, and the higher the accuracy of the determined to-be-tracked target; on the contrary, the lower the proportion, the smaller the possibility of the to-be-tracked target existing in the current frame, and the lower the accuracy of the determined to-be-tracked target.

[0049] In this way, whether the correction processing needs to be performed can be determined based on the position information, the confidence, and / or the frame serial number of the current frame, so that the correction processing is performed in time when the correction processing needs to be performed, thereby improving the accuracy of the tracking display.

[0050] Optionally, in the embodiment of the present application, the correction condition comprises:

[0051] The confidence is less than a first preset value, the frame serial number of the current frame is a multiple of a second preset value, and / or the deviation between the position information and reference position information is greater than a third preset value.

[0052] The reference position information is the position information of the to-be-tracked target in the previous frame.

[0053] That is to say, at least one of the three elements of the confidence, the frame serial number and the position information satisfies the above condition, which can be considered as satisfying the correction condition.

[0054] The following is an example.

[0055] Firstly, whether the confidence is less than the first preset value can be determined.

[0056] If yes, it indicates that the possibility of the to-be-tracked target existing in the current frame is small, so the to-be-tracked target determined in the current frame can be wrong, and therefore the correction processing needs to be performed to avoid tracking errors.

[0057] If no, it indicates that the possibility of the to-be-tracked target existing in the current frame is high, and at this time, it can be insufficient to determine only according to the confidence, so it is still necessary to continue to determine whether the frame serial number of the current frame is a multiple of the second preset value; if the frame serial number is a multiple of the second preset value, it can be understood as reaching a preset detection period, so this step is essentially to determine whether the detection period is reached.

[0058] If yes, it indicates that the detection period is reached, and then the correction processing needs to be performed, and this process can also be understood as a timed detection, that is, checking whether the tracking deviates at a time.

[0059] If no, it indicates that the detection period has not been reached, and it is still necessary to continue to determine whether the deviation between the position information and the reference position information is greater than a third preset value. This step is essentially to determine whether the position of the target to be tracked in adjacent frames has changed greatly.

[0060] If greater, it indicates that the position has changed greatly. If the target to be tracked has no error and the adjacent frames belong to the same viewpoint, the position of the target to be tracked in adjacent frames will not change greatly. Therefore, if a great change occurs, it is possible that the target to be tracked has an error, and correction processing needs to be performed.

[0061] If not greater, it indicates that the position has not changed greatly, and no correction processing is needed at this time.

[0062] In this way, it can be determined whether the target to be tracked needs to be corrected based on the position information, the confidence, and / or the frame number of the current frame in the free viewpoint video stream, thereby improving the accuracy of tracking display.

[0063] Specifically, the first preset value can be but is not limited to 0.6, and can also be 0.5, 0.7, 0.8, or 0.9, etc. It can be set according to the video content of the free viewpoint video stream, the recognition accuracy of the tracking algorithm, and other factors, which are not limited herein.

[0064] In addition, the second preset value can be but is not limited to 150, and can also be 100, 180, or 200, etc. It can be set according to the video size of the free viewpoint video stream, the processing capability of the processor, the power consumption of the processor, and the accuracy of tracking display, and other factors, which are not limited herein.

[0065] Optionally, in the embodiment of the application, the processor is configured to:

[0066] After determining the position information and the confidence of the target to be tracked in the current frame, it is determined whether the current frame is a cross-viewpoint tracking display;

[0067] If yes, it is determined that the target to be tracked needs to be corrected;

[0068] If no, it is determined that the current frame is a single-viewpoint tracking display. When it is determined that the position information, the confidence, and / or the frame number of the current frame in the free viewpoint video stream satisfies the correction condition, it is determined that the target to be tracked needs to be corrected.

[0069] That is, in addition to judging whether the current needs to be corrected based on the confidence, the position information and the frame sequence number, that is, in addition to considering the above three elements, it is also necessary to consider whether the current is cross-viewpoint tracking display. Since the positions of the target to be tracked in the frame pictures of two viewpoints may have a large deviation during cross-viewpoint tracking display, tracking errors are more likely to occur at this time. Therefore, in order to eliminate the occurrence of such problems, correction processing can be performed as long as it is determined that the current is cross-viewpoint tracking display. If the current is single-viewpoint tracking display, whether the current needs to be corrected is judged based on the confidence, the position information and the frame sequence number, thereby effectively improving the accuracy of tracking.

[0070] It should be emphasized that in the embodiments of the present application, not only single-target tracking display in a single-viewpoint scene can be realized, but also single-target tracking display in a cross-viewpoint scene can be realized, so that the intelligent device provided by the embodiments of the present application can be applied to various scenes, thereby improving the accuracy of tracking display in various scenes.

[0071] Optionally, in the embodiments of the present application, the processor is configured to:

[0072] When the deviation between the position information and the reference position information is greater than a fourth preset value, it is determined that the current is cross-viewpoint tracking display.

[0073] The reference position information is the position information of the target to be tracked in the previous frame.

[0074] Specifically, the fourth preset value can be greater than the third preset value mentioned above.

[0075] In this way, whether the current is cross-viewpoint tracking display can be determined according to the position information of the target to be tracked in the current frame and the reference position information, thereby facilitating subsequent correction processing of the target to be tracked and improving the accuracy of tracking display.

[0076] It should be emphasized that if it is determined that the current is cross-viewpoint tracking display based on the position information and the reference position information, it means that the current frame and the previous frame belong to pictures corresponding to different viewpoints. In the embodiments of the present application, whether the target to be tracked in the frame pictures of different viewpoints needs to be corrected is judged by taking the target to be tracked in the previous frame as a template, thereby realizing cross-viewpoint correction and error correction.

[0077] Optionally, in the embodiment of the present application, when the rectangular frame is used to frame the target to be tracked from the current frame, the pixel position of the center point of the rectangular frame in the current frame is determined and defined as the first pixel position, and when the rectangular frame is used to frame the target to be tracked from the previous frame, the pixel position of the center point of the rectangular frame in the previous frame is determined and defined as the second pixel position, the deviation between the position information and the reference position information is the deviation between the first pixel position and the second pixel position.

[0078] For example, if each pixel position includes the coordinate of the pixel point on the x-axis and the coordinate of the pixel point on the y-axis, the first pixel position can be represented as (x1, y1), and the second pixel position can be represented as (x2, y2), then:

[0079] The deviation between the first pixel position and the second pixel position can be represented by the following formula:

[0080] Cx = x1 - x2;

[0081] Cy = y1 - y2;

[0082] wherein x1 represents the coordinate of the pixel point on the x-axis in the first pixel position, y1 represents the coordinate of the pixel point on the y-axis in the first pixel position, x2 represents the coordinate of the pixel point on the x-axis in the second pixel position, y2 represents the coordinate of the pixel point on the y-axis in the second pixel position, Cx represents the coordinate difference of the pixel point on the x-axis between the first pixel position and the second pixel position, and Cy represents the coordinate difference of the pixel point on the y-axis between the first pixel position and the second pixel position.

[0083] Therefore, the deviation between the position information of the target to be tracked in the adjacent two frames can be determined by the deviation between the pixel positions, thereby facilitating the determination of whether the current is the cross-viewpoint tracking display.

[0084] Optionally, in order to realize the process of the correction processing, in the embodiment of the present application, the processor is configured to:

[0085] determine the feature information of the target to be tracked in the first frame of the free-viewpoint video stream;

[0086] determine the target in the current frame and the feature information of the target;

[0087] when there are multiple targets in the current frame, calculate the feature similarity between each target and the target to be tracked according to the feature information of the target to be tracked and the feature information of each target;

[0088] take the target with the highest feature similarity as the target to be tracked after the correction processing.

[0089] Specifically, in the determination of the feature information, a target detection algorithm can be used, which can be any algorithm known to those skilled in the art and is not limited herein.

[0090] In the calculation of the feature similarity, any method known to those skilled in the art can be used, which is not limited herein.

[0091] It should be noted that, in the correction process, the determination of the feature information can be regarded as a target detection process, and the calculation of the feature similarity and the correction of the target to be tracked can be regarded as a target re-identification process.

[0092] Therefore, in the embodiments of the present application, the target detection process and the target re-identification process can be combined to determine the accuracy of the tracked object in the tracking display process and to re-track in the cross-viewpoint tracking display, so as to ensure the accuracy of the tracking display.

[0093] It should be noted that, in the embodiments of the present application, the target detection process is intended to determine the positions of the targets in the current frame, and a rectangular frame can be used to mark the positions. Taking a player as an example, since the current target detection algorithm is capable of distinguishing multiple categories, the target detection process is improved to detect the category of the player, i.e., to detect the feature information of the player.

[0094] In addition, due to the existence of interference factors such as spectators in the stadium, the aspect ratio of the rectangular frame is set to a threshold value to distinguish the player from the spectator. That is, if the aspect ratio of the rectangular frame is greater than the threshold value, it means that the rectangular frame marks the spectator rather than the player, thereby eliminating the interference of the interference factors.

[0095] Optionally, in the embodiments of the present application, the instruction is issued when the user selects the target to be tracked from the picture of the first frame in the display of the first frame of the free-viewpoint video stream.

[0096] That is, in the determination of the target to be tracked, the following method can be used:

[0097] When the first frame of the free-viewpoint video stream is displayed, if multiple targets exist in the picture of the first frame, the user can select one of the multiple targets as the target to be tracked, so that the processor can receive the instruction and determine the target to be tracked based on the instruction and start the tracking display.

[0098] Specifically, in the embodiments of the present application, when the user selects the target to be tracked, the following methods can be used:

[0099] Manner 1: when the display has a touch function, the user selects the target to be tracked by touching a target displayed on the interface of the display;

[0100] Manner 2: when the display has a function button, the user selects the target to be tracked by the function button;

[0101] Regardless of the above-mentioned manners 1 and 2, the user can directly select the target to be tracked or frame the target to be tracked with a rectangular frame.

[0102] Of course, in actual cases, the user can select the target to be tracked not only from the first frame but also in the following manners:

[0103] Before the display displays the first frame, the targets existing in the first frame are identified, and the identified results are displayed in the form of a list;

[0104] The user can select through the list;

[0105] After the user selects, the first frame is displayed.

[0106] Specifically, the manner in which the user selects the target to be tracked can be set according to actual needs, which is not limited herein.

[0107] Optionally, in the embodiment of the application, the processor is configured to:

[0108] When the rectangular frame is used to frame the target to be tracked from the picture of the first frame, the background filtering processing is performed on the framed area.

[0109] That is to say, when the target to be tracked is framed from the picture of the first frame, the framed area includes the target to be tracked and background information, and if the background information is too much, it may affect the identification and tracking of the target to be tracked, so the background filtering processing can be performed on the framed area to filter out useless background information and enhance the information of the target to be tracked, thereby improving the accuracy of identification and ultimately improving the accuracy of tracking display.

[0110] Optionally, in the embodiment of the application, when the rectangular frame is used to frame the target to be tracked from the picture of the first frame, the processor is configured to:

[0111] Based on the Siam tracking algorithm, the position information of the rectangular frame in the picture of the first frame is determined;

[0112] The feature information of the target to be tracked is determined.

[0113] Wherein, when the feature information of the target to be tracked is determined, the following manners can be adopted:

[0114] Manner 1: determined by using a target detection algorithm;

[0115] Mode 2: Taking the free-viewpoint video stream as a basketball game video as an example, when the to-be-tracked target is a player, the feature information of the to-be-tracked target can be the number on the jersey, and the number can be manually determined by the user; or when the to-be-tracked target is a basketball, the feature information of the to-be-tracked target can be a circle, and the circle can also be manually determined by the user.

[0116] Of course, in addition to the above-mentioned mode 1 and mode 2, the method for determining the feature information of the to-be-tracked target can also be other methods that can achieve this function, which is not limited herein.

[0117] The working process of the above-mentioned intelligent device provided by the embodiment of the application will be described below in combination with specific embodiments.

[0118] In combination with Figure 7 Taking that the user frames the to-be-tracked target in the picture of the first frame with a rectangular frame as an example, for the second frame and each frame thereafter in the free-viewpoint video stream, the Siam tracking algorithm can be used to determine the rectangular frame (i.e., the rectangular frame framing the to-be-tracked target), the center point position of the rectangular frame, and the confidence level in each frame from the second frame onwards according to the to-be-tracked target.

[0119] The following process is performed for each frame from the second frame onwards:

[0120] S701, determining whether the confidence level of the current frame is greater than a first preset value; if yes, performing S702; if no, performing S705;

[0121] S702, determining whether the frame sequence number of the current frame is a multiple of a second preset value; if no, performing S703; if yes, performing S705;

[0122] S703, determining whether the offset between the center point pixel position of the rectangular frame in the current frame and the center point pixel position of the rectangular frame in the previous frame is greater than a fourth preset value; if yes, determining that the current is a cross-viewpoint tracking display, and performing S705; if no, determining that the current is a single-viewpoint tracking display, and performing S704;

[0123] S704, determining whether the offset between the center point pixel position of the rectangular frame in the current frame and the center point pixel position of the rectangular frame in the previous frame is greater than a third preset value; if yes, performing S705; if no, performing S709;

[0124] The third preset value is less than the fourth preset value.

[0125] S705, determining that the to-be-tracked target in the current frame needs to be corrected;

[0126] S706, determine feature information of the target to be tracked in the first frame of the free-viewpoint video stream, and determine the target in the current frame and the feature information of the target;

[0127] S707, when there are multiple targets in the current frame, calculate feature similarity between each target and the target to be tracked according to the feature information of the target to be tracked and the feature information of each target, and take the target with the highest feature similarity as the target to be tracked after correction processing; at the same time, the target to be tracked after correction processing is framed by a rectangular frame, and the center point position of the rectangular frame is determined;

[0128] Note that the rectangular frame in the current frame can be deleted before, during or after the correction processing, so as to avoid interference when the target to be tracked after correction processing is framed by a rectangular frame.

[0129] S708, track and display based on the target to be tracked after correction processing; end the process;

[0130] S709, track and display based on the current target to be tracked.

[0131] Specifically, based on the working process of the above-mentioned intelligent device, the result of the tracking display can be seen from Figure 8 As shown in the figure, Fig. (a) and Fig. (b) are the pictures corresponding to one of the viewpoints, wherein Fig. (a) is the picture of the first frame, and Fig. (b) is the picture of the 50th frame; Fig. (c) and Fig. (d) are pictures corresponding to another viewpoint, wherein Fig. (c) is the picture of the 100th frame, and Fig. (d) is the picture of the 150th frame;

[0132] From the four pictures, it can be found that the target framed by the rectangular frame is the target to be tracked and is the same, so there is no tracking error problem when tracking and displaying, and the target to be tracked remains consistent, thereby realizing accurate tracking and display.

[0133] Based on the same inventive concept, the embodiment of the application provides a target tracking display method, and the implementation principle of the control method is similar to that of the aforementioned intelligent device. The specific implementation of the control method can be seen from the specific embodiment of the aforementioned intelligent device, and the repeated parts will not be described again.

[0134] Specifically, the target tracking display method provided by the embodiment of the application, as shown in Figure 9 may include:

[0135] S901, in response to an instruction to track and display the target to be tracked;

[0136] S902, processing the to-be-tracked target in each frame in the free-viewpoint video stream as follows: when rectification processing is needed for the to-be-tracked target, performing the rectification processing.

[0137] Optionally, in the embodiment of the present application, when it is determined that the to-be-tracked target needs to be rectified, the method specifically comprises:

[0138] determining the position information and the confidence of the to-be-tracked target in the current frame;

[0139] when the position information, the confidence, and / or the frame sequence number of the current frame in the free-viewpoint video stream meets the preset rectification condition, it is determined that the to-be-tracked target needs to be rectified.

[0140] Optionally, in the embodiment of the present application, after the position information and the confidence of the to-be-tracked target in the current frame are determined, the method further comprises:

[0141] determining whether the current frame is a cross-viewpoint tracking display;

[0142] if yes, it is determined that the to-be-tracked target needs to be rectified;

[0143] if no, it is determined that the current frame is a single-viewpoint tracking display, and when the position information, the confidence, and / or the frame sequence number of the current frame in the free-viewpoint video stream meets the rectification condition, it is determined that the to-be-tracked target needs to be rectified.

[0144] Optionally, in the embodiment of the present application, when it is determined that the current frame is a cross-viewpoint tracking display, the method specifically comprises:

[0145] when the deviation between the position information and the reference position information is greater than a fourth preset value, it is determined that the current frame is a cross-viewpoint tracking display;

[0146] wherein the reference position information is the position information of the to-be-tracked target in the previous frame.

[0147] Optionally, in the embodiment of the present application, when the to-be-tracked target is rectified, the method specifically comprises:

[0148] determining the feature information of the to-be-tracked target in the first frame of the free-viewpoint video stream;

[0149] determining the target in the current frame and the feature information of the target;

[0150] when there are multiple targets in the current frame, calculating the feature similarity between each target and the to-be-tracked target according to the feature information of the to-be-tracked target and the feature information of each target;

[0151] taking the target with the highest feature similarity as the rectified to-be-tracked target.

[0152] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A smart device, characterized in that, include: The monitor is configured to display images; The processor is configured as follows: Responding to user commands to display the tracking target; The display is controlled to perform tracking display when displaying a free-viewpoint video stream; In the aforementioned tracking display, the target to be tracked in each frame is obtained after the correction process is performed when the target to be tracked needs to be corrected. The processor is configured as follows: Determine the position information and confidence level of the target to be tracked in the current frame; When it is determined that the location information, the confidence level, and / or the frame number of the current frame in the free viewpoint video stream meet the preset correction conditions, it is determined that the target to be tracked needs to undergo the correction process; wherein, the confidence level is the proportion of the target to be tracked in the current frame; The correction conditions include: The confidence level is less than a first preset value, the frame number of the current frame is a multiple of a second preset value, and / or the deviation between the location information and the reference location information is greater than a third preset value; The reference position information is: the position information of the target to be tracked in the previous frame; The processor is configured as follows: After determining the position information and confidence level of the target to be tracked in the current frame, it is determined whether the current display is a cross-viewpoint tracking display; If so, then it is determined that the target to be tracked needs to undergo the corrective processing. If not, then it is determined that the current tracking display is a single viewpoint. When it is determined that the position information, the confidence level, and / or the frame number of the current frame in the free viewpoint video stream meet the correction conditions, it is determined that the target to be tracked needs to be corrected. The processor is configured as follows: When the deviation between the location information and the reference location information is greater than a fourth preset value, it is determined that the current state is the cross-viewpoint tracking display; The reference position information is: the position information of the target to be tracked in the previous frame; the fourth preset value is greater than the third preset value.

2. The intelligent device as described in claim 1, characterized in that, When a rectangle is used to frame the target to be tracked from the current frame, and the pixel position of the center point of the rectangle in the current frame is determined and defined as the first pixel position, and when a rectangle is used to frame the target to be tracked from the previous frame, and the pixel position of the center point of the rectangle in the previous frame is determined and defined as the second pixel position, the deviation between the position information and the reference position information is the deviation between the first pixel position and the second pixel position.

3. The intelligent device as described in claim 1, characterized in that, The processor is configured as follows: Determine the feature information of the target to be tracked in the first frame of the free viewpoint video stream; Determine the target in the current frame and the feature information of the target; When multiple targets exist in the current frame, the feature similarity between each target and the target to be tracked is calculated based on the feature information of the target to be tracked and the feature information of each target. The target with the highest feature similarity is selected as the target to be tracked after the correction process.

4. The intelligent device as described in claim 1, characterized in that, The instruction is given by the user when the first frame of the free-viewpoint video stream is displayed on the monitor, and the user selects the target to be tracked from the first frame.

5. The intelligent device as described in claim 4, characterized in that, The processor is configured as follows: When the target to be tracked is outlined from the first frame using a rectangular bounding box, the outlined area is subjected to background filtering.

6. A method for target tracking and display, characterized in that, include: In response to a command to display the tracking information of the target being tracked; The target to be tracked in each frame of the free viewpoint video stream is processed as follows: when correction processing of the target to be tracked is required, the correction processing is performed; Determine if corrective action is needed for the target to be tracked, including: Determine the position information and confidence level of the target to be tracked in the current frame; When it is determined that the location information, the confidence level, and / or the frame number of the current frame in the free viewpoint video stream meet the preset correction conditions, it is determined that the target to be tracked needs to undergo the correction process; wherein, the confidence level is the proportion of the target to be tracked in the current frame; The correction conditions include: The confidence level is less than a first preset value, the frame number of the current frame is a multiple of a second preset value, and / or the deviation between the location information and the reference location information is greater than a third preset value; The reference position information is: the position information of the target to be tracked in the previous frame; After determining the location information and confidence level of the target to be tracked in the current frame, the following steps are also included: Determine if the current display is a cross-viewpoint tracking display; If so, then it is determined that the target to be tracked needs to be corrected. If not, then it is determined that the current tracking display is a single viewpoint. When the position information, confidence level, and / or the frame number of the current frame in the free viewpoint video stream meet the correction conditions, it is determined that the target to be tracked needs to be corrected. The current display is determined to be cross-viewpoint tracking, including: When the deviation between the location information and the reference location information is greater than the fourth preset value, it is determined that the current display is a cross-viewpoint tracking display; The reference position information is: the position information of the target to be tracked in the previous frame; the fourth preset value is greater than the third preset value.

Images