Monitoring image tracking method and device, terminal equipment and image collector

Abstract

The invention relates to a monitoring image tracking method and device, terminal equipment and an image acquirer, after the image acquirer performs image acquisition to obtain image frame data, the image frame data is transmitted to the terminal equipment to be analyzed, and interaction contacts corresponding to all objects to be selected in the image frame data are obtained and displayed. In the embodiment of the invention, a user only needs to perform touch interaction on the interaction touch piece and send the target identity information of the to-be-selected object corresponding to the interaction touch piece to the image collector, so that the to-be-selected object can be determined as the target tracking object, the target tracking object is tracked, and the target image data frame corresponding to the target tracking object is returned to the terminal equipment. According to the scheme, a user can freely select and switch the target tracking object on the terminal equipment, so that the tracking target of the image collector is actively selected or changed.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a surveillance image tracking method, device, terminal equipment and image collector. Background Art

[0002] With the development of science and technology, cameras are becoming more and more intelligent. Combined with current computer vision technology, cameras already have AI (Artificial Intelligence) tracking capabilities, which can identify targets such as people, cars, pets, etc. in the picture, and adjust the field of view direction through the gimbal to finally lock the target object.

[0003] However, in the related art, the user cannot actively select or change the tracking target of the camera. Summary of the invention

[0004] Based on this, it is necessary to provide a monitoring image tracking method, apparatus, terminal device and image collector to solve the problem that the user cannot actively select or change the tracking target of the camera.

[0005] The present application provides a monitoring image tracking method, including: receiving image frame data collected and sent by an image collector; parsing and displaying interactive contacts corresponding to each object to be selected in the image frame data; associating the interactive contacts with identity information of the object to be selected; responding to the interactive action of the interactive contacts, sending the target identity information of the object to be selected corresponding to the interactive contacts to the image collector; the target identity information is used to determine the target tracking object; receiving the target image frame data tracked and sent by the image collector to the target tracking object.

[0006] In the above scheme, after the image collector acquires the image frame data through image acquisition, it is transmitted to the terminal device for parsing, and the interactive touch elements corresponding to each selected object in the image frame data are obtained and displayed. The user only needs to touch the interactive touch elements and send the target identity information of the selected object corresponding to the interactive touch elements to the image collector, so that the selected object can be determined as the target tracking object, track it, and return the target image data frame corresponding to the target tracking object to the terminal device. Through this scheme, the user can freely select and switch the target tracking object on the terminal device, thereby actively selecting or changing the tracking target of the image collector.

[0007] The present application also provides a monitoring image tracking method, including: collecting image frame data and sending it to a terminal device; the image frame data is used by the terminal device to parse and obtain interactive contacts corresponding to each object to be selected; the interactive contacts are associated with the identity information of the object to be selected; receiving the target identity information sent by the terminal device to determine the target tracking object; the target identity information is sent by the terminal device in response to the interactive action of the interactive contact; tracking and collecting the target tracking object to obtain target image frame data, and sending it to the terminal device.

[0008] The present application also provides a monitoring image tracking device, including: an image frame receiving module, used to receive image frame data collected and sent by an image collector; an image frame parsing module, used to parse and display interactive contacts corresponding to each object to be selected in the image frame data; the interactive contacts are associated with the identity information of the object to be selected; an interactive response module, used to respond to the interactive action of the interactive contacts, and send the target identity information of the object to be selected corresponding to the interactive contacts to the image collector; the target identity information is used to determine the target tracking object; a target image frame receiving module, used to receive the target image frame data tracked and sent by the image collector to the target tracking object.

[0009] The present application also provides a monitoring image tracking device, including: an image frame acquisition module, used to acquire image frame data and send it to a terminal device; the image frame data is used by the terminal device to parse and obtain interactive contacts corresponding to each object to be selected; the interactive contacts are associated with the identity information of the object to be selected; a target identity receiving module, used to receive the target identity information sent by the terminal device and determine the target tracking object; the target identity information is sent by the terminal device in response to the interactive action of the interactive contact; a target tracking module, used to track and acquire the target tracking object to obtain target image frame data, and send it to the terminal device.

[0010] The present application also provides a terminal device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned monitoring image tracking method when executing the computer program.

[0011] The present application also provides an image collector, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned monitoring image tracking method when executing the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of an application environment of a monitoring image tracking method in one embodiment of the present application;

[0013] Figure 2This is a flow chart of a monitoring image tracking method in one embodiment of the present application;

[0014] Figure 3 This is a flow chart of a monitoring image tracking method in another embodiment of the present application;

[0015] Figure 4 This is a schematic diagram of a canvas step length determination process in one embodiment of the present application;

[0016] Figure 5 This is a schematic diagram of a canvas step length determination process in another embodiment of the present application;

[0017] Figure 6 This is a schematic diagram of a frame insertion process in one embodiment of the present application;

[0018] Figure 7 This is a flow chart of a monitoring image tracking method in another embodiment of the present application;

[0019] Figure 8 This is an interactive schematic diagram of a monitoring image tracking method in one embodiment of the present application;

[0020] Fig. 9 This is a schematic diagram of video screen display in one embodiment of the present application;

[0021] Fig.10 This is a schematic diagram of the structure of a monitoring image tracking device in one embodiment of the present application;

[0022] Fig.11 This is a schematic diagram of the structure of a monitoring image tracking device in another embodiment of the present application;

[0023] Fig.12 This is a schematic diagram of the structure of a monitoring image tracking device in another embodiment of the present application;

[0024] Fig.13 This is a schematic diagram of the internal structure of a terminal device in one embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] The monitoring image tracking method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the terminal device 102 communicates with the image collector 104 through the network. The terminal device 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices, and the image collector 104 may be a camera with AI functions, etc., to realize the video surveillance function, which is not specifically limited.

[0027] See also Figure 2 , taking the monitoring image tracking method applied to a terminal device as an example for explanation, the method includes step 202, step 204, step 206 and step 208.

[0028] Step 202: receiving image frame data collected and sent by an image collector.

[0029] Specifically, the image frame data is the data that transmits the collected video image and its related status parameters in the form of data frames during monitoring. The specific type of image frame data is not unique. In a more detailed embodiment, the image frame data includes SEI, that is, Supplemental Enhancement Information, which can carry supplementary information related to encoding and display, and is essentially a type of network access unit defined in the video codec standard (h.264, h.265).

[0030] In actual application scenarios (such as security monitoring), real-time picture acquisition is performed through an image collector to obtain a code stream of the monitored scene. Specifically, the image collector can acquire two code streams (i.e., YUV streams), one of which is used for real-time streaming transmission and sent to the terminal device for display so that the user can view it on the terminal device. The other is used for AI analysis. The corresponding candidate objects in the code stream can be determined through AI algorithm analysis, and then the identity information (which may also include position information in other embodiments) is configured for each candidate object, and the position information and / or identity information is packaged through SEI information to obtain image frame data and transmit it to the terminal device.

[0031] It should be pointed out that the type of image collector is not unique. In one embodiment, the image collector may be a camera, and in another embodiment, the image collector may be an image sensor, etc. There is no specific limitation as long as it has image acquisition and AI analysis functions.

[0032] Step 204: parse and display the interactive touch items corresponding to the selected objects in the image frame data.

[0033] Specifically, the interactive touch is associated with the identity information of the object to be selected. The object to be selected is the object waiting to be selected as the tracking target of the image collector. According to different actual needs, the type of the object to be selected will also be different. It can be a human body, an animal or a plant, or a vehicle, etc., without specific limitation. When watching the real-time stream at the terminal device, the terminal device will display the identity information and location information in the image frame data to the user through the interactive touch.

[0034] It should be noted that the specific type of interactive touch is not unique, and the form of the interactive touch may vary depending on the type of terminal device. For example, in one embodiment, a rectangular frame or a frame of other shapes may be used as an interactive touch to frame the object to be selected. In another embodiment, it may be implemented by configuring touch buttons around the object to be selected, etc., without specific limitation.

[0035] Step 206, in response to the interactive action of the interactive touch element, the target identity information of the selected object corresponding to the interactive touch element is sent to the image collector.

[0036] Specifically, the target identity information is used to determine the target tracking object. After the terminal device parses and displays the interactive touches corresponding to each selected object, the user can select the object that currently needs to be tracked by the image collector on the display interface of the terminal device. According to actual needs, the user only needs to touch the interactive touch of the object to be tracked on the display interface, and the terminal device can respond to the interactive action and transmit the identity information associated with the interactive touch as the target identity information to the image collector.

[0037] Step 208: receiving target image frame data sent by the image collector after tracking the target tracking object.

[0038] Specifically, the target tracking object is the target tracking object that needs to be tracked by the image collector at present. After the image collector receives the target identity information, the target tracking object can be determined by the target identity information, and the target tracking object is tracked, and the target image frame data in the process of tracking the target tracking object is collected and sent to the terminal device, thereby completing the operation of actively selecting or changing the tracking target of the image collector.

[0039] More specifically, in one embodiment, the image collector tracks the target tracking object by adjusting the field of view (FOV) of the image collector according to the corresponding movement or change of the target tracking, and monitoring the changes of the target tracking object within the monitoring range of the image collector in real time.

[0040] In the above scheme, after the image collector acquires the image frame data through image acquisition, it is transmitted to the terminal device for parsing, and the interactive touch elements corresponding to each selected object in the image frame data are obtained and displayed. The user only needs to touch the interactive touch elements and send the target identity information of the selected object corresponding to the interactive touch elements to the image collector, so that the selected object can be determined as the target tracking object, track it, and return the target image data frame corresponding to the target tracking object to the terminal device. Through this scheme, the user can freely select and switch the target tracking object on the terminal device, thereby actively selecting or changing the tracking target of the image collector.

[0041] See also Figure 3 In one embodiment, after step 208 , the method further includes steps 302 and 304 .

[0042] Step 302: Determine the canvas step length between adjacent video images to be displayed according to adjacent first target image frame data and second target image frame data.

[0043] Step 304 : inserting and displaying a differential video image between a first video image rendered by the first target image frame data and a second video image rendered by the second target image frame data according to the canvas step size.

[0044] Specifically, the differentiated video image is determined based on the first video image. In actual application scenarios, the encoding frame rate of the image collector is certain and is often less than the refresh rate of the terminal device (such as a mobile phone). For example, in a more detailed embodiment, the encoding frame rate of the more common 4K camera is generally 15 frames, so the effect it can achieve is a 15-frame / second screen display, while a continuous screen that is imperceptible to the human eye requires at least 24 frames / second. If the image is tracked at the default encoding frame rate of the image collector, it is easy to cause the screen to freeze.

[0045] The solution of this embodiment performs frame interpolation processing between the adjacent first target image frame data and the second target image frame data collected by the image collector, and inserts and displays a differentiated video image within the display time interval from the first image frame data to the second image frame data, so that the transition between the rendering display of the first image frame data and the rendering display of the second image frame data is smoother, thereby reducing the possibility of screen freezes.

[0046] The solution of this embodiment first analyzes the adjacent first target image frame data and second target image frame data to determine the canvas step length that needs to be adjusted between adjacent video images after inserting the differential video image between the first target image frame data and the second target image frame data. Then, during the display process, after the first image corresponding to the first target image frame data is displayed, the canvas is adjusted according to the canvas step length, and the differential video image is inserted and displayed. Subsequently, the canvas is adjusted according to the canvas step length in turn, and each differential video image is inserted and displayed until all the differential video images are inserted and displayed, and the second image corresponding to the second target image frame data is displayed.

[0047] Through this solution, without changing the display time of the first video image rendered by the first target image frame data and the second video image rendered by the second target image frame data, an additional differentiated video image is inserted in the display time interval between the two, and the screen display limited by the encoding frame rate of the image acquisition device is converted to a higher frame rate display, thereby reducing the possibility of screen freezes.

[0048] See also Figure 4 In one embodiment, step 302 includes step 402 and step 404.

[0049] Step 402: parse adjacent first target image frame data and second target image frame data to determine first position parameters and second position parameters of the target tracking object.

[0050] Step 404: Determine the canvas step length between adjacent video images to be displayed according to the first position parameter and the second position parameter.

[0051] Specifically, the first position parameter is the position parameter of the target tracking object in the video image corresponding to the first target image frame data. The second position parameter is the position parameter of the target tracking object in the video image corresponding to the second target image frame data. The video image to be displayed includes the first video image, the second video image, and a differential video image therebetween. The adjacent video images may be the first video image and the differential video image, two differential video images, or the differential video image and the second video image.

[0052] In the solution of this embodiment, the image frame data carries information related to the position parameters, so after the terminal device obtains the first target image frame data and the second target image frame data, the first position parameter and the second position parameter can be obtained by parsing them.

[0053] In actual application scenarios, the first position parameter and the second position parameter are combined to show that the target tracking object changes position from the first position parameter to the second position parameter. Therefore, in actual canvas step analysis, in order to show the movement of the target tracking object from the first position parameter to the second position parameter as much as possible, the first position parameter and the second position parameter are combined to analyze and determine the canvas step between each adjacent video image.

[0054] It should be noted that the specific type of position parameters is not unique, and in a more detailed embodiment, the position parameters are all represented by coordinate parameters, that is, when the image collector collects video streams, coordinate parameters are configured for each candidate object as position parameters.

[0055] It can be understood that in one embodiment, since the target tracking object often occupies a certain area in the video image, in order to accurately represent the position information of the target tracking object in the video screen, the target tracking object can be selected in the form of a rectangular frame, and then the vertex coordinates of the rectangular frame are used to represent the position information of the target tracking object.

[0056] Furthermore, in one embodiment, in order to reduce the amount of calculation when calculating the canvas step length, the coordinates of any two diagonal vertices of the rectangular frame can also be used to represent the position information of the target tracking object, so as to perform the canvas step length calculation. For example, the coordinates of the upper left and lower right vertices of the rectangular frame can be used to represent the position information of the target tracking object, or the coordinates of the lower left and upper right vertices of the rectangular frame can be used to represent the position information of the target tracking object.

[0057] See also Figure 5 In one embodiment, step 404 includes step 502 , step 504 , and step 506 .

[0058] Step 502, obtaining a target refresh rate and an encoding frame rate of an image collector.

[0059] Step 504, determining the number of required differential video images according to the encoding frame rate and the target refresh rate;

[0060] Step 506: Determine the canvas step length between adjacent video images to be displayed according to the quantity, the first position parameter, and the second position parameter.

[0061] Specifically, the target refresh rate is the canvas refresh rate that is expected to be achieved after interpolation of adjacent target image frame data. The encoding frame rate is the number of image frame data that can be transmitted by the image collector within one second.

[0062] It should be noted that the target refresh rate is not unique and can be determined in combination with the type of terminal device in the actual scenario. For example, in a more detailed embodiment, the terminal device is a mobile phone and the target refresh rate can be set to 60fps (Frame Per Second).

[0063] To facilitate understanding of the solution of this embodiment, an explanation is given by taking an example in which the position parameters include the coordinates of the diagonal vertices of a rectangular box in which the target tracking object is selected, the target refresh rate is 60fps, and the encoding frame rate is 15 frames.

[0064] First, the image collector performs image acquisition and encoding processing, and sends the first target image frame data and the second target image frame data to the terminal device in sequence. The first target image frame data and the second target image frame data are first cached in a buffer, and the terminal device parses the first target image frame data to obtain a first position parameter, that is, the coordinates of the two diagonal vertices A1 and B1, which are (XA1, YA1) and (XB1, YB1) respectively; and parses the second target image data to obtain a second position parameter, that is, the coordinates of the two diagonal vertices A2 and B2, which are (XA2, YA2) and (XB2, YB2) respectively.

[0065] Then, the step length is determined by combining the first position parameter and the second position parameter. Accordingly, since there are two vertices with the same position parameter at this time (both are upper left and lower right vertices, or both are lower left and upper right vertices), the step lengths of the two vertices need to be calculated separately. According to XA2-XA1 and YA2-YA1, the canvas step length corresponding to vertex A1 to vertex A2 is determined. According to XB2-XB1 and YB2-YB1, the canvas step length corresponding to vertex B1 to vertex B2 is determined. Through the above-determined canvas step length, three differentiated video images are finally inserted between the first video image and the second video image, so that the target refresh rate can be increased to 60fps (frames per second) based on the original 15 frames.

[0066] In one embodiment, the canvas step sizes between adjacent video images to be displayed are all consistent.

[0067] Specifically, in this embodiment, the canvas step lengths adjusted between each adjacent video screen are set to be the same, so as to improve the insertion efficiency of the differentiated video image. Similarly, taking the above-mentioned target refresh rate of 60fps and the encoding frame rate of 15 frames as an example, if the canvas step length needs to be set to be the same, XA2-XA1 and YA2-YA1, as well as XB2-XB1 and YB2-YB1 need to be equally divided. When the number of inserted differentiated video images is three, the position difference between the first video image and the second video image is divided into four equal parts. Therefore, in actual scenarios, the canvas step length can be set to ((XA2-XA1) / 4, (YA2-YA1) / 4), and ((XB2-XB1) / 4, (YB2-YB1) / 4).

[0068] It is understood that in other embodiments, in order to achieve a better smooth transition effect, the canvas step lengths may be set to be different. For example, the canvas step length may be set to be gradually increased or gradually decreased, depending on actual needs.

[0069] See also Figure 6 In one embodiment, step 304 includes step 602 , step 604 , step 606 and step 608 .

[0070] Step 602: Determine a preset refresh time interval according to the encoding frame rate of the image collector.

[0071] Step 604 , displaying a first video image rendered by the first target image frame data at an initial moment.

[0072] Step 606 , after adjusting the canvas according to the canvas step size every time the preset refresh time interval is increased, the first video image is displayed as a differential video image.

[0073] Step 608 : If the inserted differential video images reach the required number, display the second video image rendered by the second target image frame data.

[0074] Specifically, the preset refresh time interval is the preset display time interval between adjacent video images. For ease of understanding, taking the target refresh rate of 60fps and the encoding frame rate of 15 frames as an example, if the image collector needs to track the target tracking object at 10 frames (15pfs), the time interval between adjacent target image frame data can reach 10 / 15, which is about 66.67 milliseconds (ms). In actual application scenarios, if the target refresh rate of 60fps needs to be achieved, three differentiated video images need to be inserted between the first target image frame data and the second target image frame data. Accordingly, the display time interval between each video image can be set to 66.67 / 4, which is about 16.66 milliseconds.

[0075] Therefore, after the terminal device receives the first image frame data, it is transmitted to the hardware decoder and rendered, and the player of the terminal device notifies the upper layer UI (User Interface) to start rendering the first image frame data. At time t=0.0ms, the screen of the terminal device displays the first image; at time t=16.66ms, since the player has no data to render at this time, the screen still displays the first video image corresponding to the first image frame data, and the terminal device will adjust the canvas by one canvas step; at time t=16.66ms+16.66ms, since the player has no data to render at this time, the screen still displays the first video image corresponding to the first image frame data, and the terminal device will continue to adjust the canvas by one canvas step; at time t=16.66ms+16.66ms+16.66ms, the player also has no data to render at this time, and the screen still displays the first video image corresponding to the first image frame data, and the terminal device will continue to adjust the canvas by one canvas step; and at time t=16.66ms+16.66ms+16.66ms+16.66ms, since the player starts to render the second image frame data at this time, the terminal device will adjust the canvas to display the second video image rendered by the second target image frame data.

[0076] In one of the embodiments, it also includes: if during the process of inserting the differential video image, the third target image frame data is received, ending the insertion of the differential video image between the first video image and the second video image; redetermining the canvas step size according to the second target image frame data and the third target image frame data; and inserting and displaying the differential video image between the second video image and the third video image rendered by the third target image frame data according to the redetermined canvas step size.

[0077] Specifically, the differentiated video image is determined according to the second video image. In the solution of this embodiment, when the differentiated video image is inserted between the first target image frame data and the second target image frame data, it is also monitored whether the terminal device receives new target image frame data. In the case of receiving new target image frame data, the insertion of the current differentiated video image will be terminated and the insertion operation of the next differentiated video image will be executed. Through this solution, data processing delay can be effectively avoided and image tracking reliability can be improved.

[0078] In one of the embodiments, tracking the target tracking object includes scaling the target tracking object or outputting position information of the target tracking object.

[0079] Specifically, when the image collector tracks the target tracking object, it can output the position information corresponding to the target tracking object to inform relevant personnel. In another embodiment, the target tracking object can also be highlighted, for example, the part of the target tracking object in the video screen is zoomed, etc., which is not specifically limited.

[0080] It should be noted that in actual application scenarios, the method of scaling the target tracking object is not unique. In a more detailed embodiment, it can be implemented by a scaling algorithm (such as a Bezier curve), which can be selected in combination with actual needs. The scaling multiple of the target tracking object is not unique, and it can be set in combination with actual needs. For example, the target tracking object is magnified three times for display, and the other selected objects are not adjusted.

[0081] See also Figure 7 , a monitoring image tracking method includes step 702, step 704 and step 706.

[0082] Step 702: collect image frame data and send it to the terminal device.

[0083] Step 704: Receive target identity information sent by the terminal device and determine the target tracking object.

[0084] Step 706: Track and capture the target tracking object to obtain target image frame data, and send it to the terminal device.

[0085] Specifically, the image frame data is used by the terminal device to parse and obtain the interactive touches corresponding to each object to be selected; the interactive touches are associated with the identity information of the object to be selected; and the target identity information is sent by the terminal device in response to the interactive action of the interactive touches.

[0086] In actual application scenarios (such as security monitoring), the image collector is used to collect real-time images and obtain the code stream of the monitored scene. Specifically, the image collector can collect and obtain two code streams, one of which is used for real-time streaming and sent to the terminal device for display so that the user can view it on the terminal device. The other is used for AI analysis. Through AI algorithm analysis, the corresponding candidate objects in the code stream can be determined, and then the relevant information of each candidate object is configured to obtain image frame data and transmit it to the terminal device.

[0087] According to different actual needs, the type of the selected object will also be different, which can be human body, animal or plant, etc., without specific limitation. When watching the real-time stream at the terminal device, the terminal device will display the relevant configuration information in the image frame data to the user through the interactive touch.

[0088] After the terminal device parses and displays the interactive touches corresponding to each selected object, the user can select the object that needs to be tracked by the image collector on the display interface of the terminal device. According to actual needs, the user only needs to touch the interactive touch of the object to be tracked on the display interface, and the terminal device can respond to the interactive action and transmit the identity information associated with the interactive touch as the target identity information to the image collector.

[0089] After the image collector receives the target identity information, it can determine the target tracking object based on the target identity information, track the target tracking object, collect the target image frame data during the tracking process of the target tracking object, and send it to the terminal device, thereby completing the active selection or change of the tracking target of the image collector.

[0090] In one of the embodiments, after collecting the image frame data, the method further includes: tracking the selected objects in the image frame data according to a preset priority order.

[0091] Specifically, in the scheme of this embodiment, when the image collector obtains the video code stream for AI analysis, it will also combine the AI ​​algorithm and its own preset priority order to select the candidate object with the highest priority in the video image, and track the candidate object, such as scaling the candidate object, etc., without specific limitation.

[0092] It should be noted that the type of preset priority order is not unique. In one embodiment, it can be the type priority of the selected object, for example, the human body priority is higher than the animal body priority, and the animal body priority is higher than the moving object priority, etc. In another embodiment, it can also be the distance priority between the selected object and the image collector, for example, the closer to the image collector, the higher the priority.

[0093] In one of the embodiments, collecting image frame data includes: obtaining video code stream data and determining objects to be selected; configuring identity information and location information for each object to be selected to obtain image frame data.

[0094] Specifically, during the acquisition of image data frames, the image collector acquires two code streams, one of which is used for real-time streaming and sent to the terminal device for display so that the user can view it on the terminal device. The other is used for AI analysis. Through AI algorithm analysis, the corresponding candidate objects in the code stream can be determined, and then the identity information and location information of each candidate object are configured, and the location information and / or identity information are packaged through SEI information to obtain image frame data and transmit it to the terminal device.

[0095] In order to facilitate understanding of the technical solution of the present application, the present application is explained below in conjunction with a more detailed embodiment: The image collector is a camera, whose image encoding frame rate is 15 frames, and the terminal device is a mobile phone, and the required target refresh rate is 60fps.

[0096] Please refer to Figure 8 First, the camera collects images of the monitored space and obtains two code streams, one of which is used for real-time streaming and sent to the mobile phone (the relevant APP in the mobile phone) for display so that the user can view it on the terminal device. The other is used for AI analysis. The AI ​​algorithm analysis can determine the corresponding candidate objects in the code stream, track each candidate object according to the preset priority order of the camera itself, and use the candidate object with the highest priority as the target tracking object. After scaling, it is encoded and transmitted to the mobile phone in the form of a target image data frame. The camera selects each candidate object through a rectangular frame. During the encoding process, the coordinate parameters of the diagonal vertices of the rectangular frame and the identity information of each candidate object are configured in the target image frame data for transmission.

[0097] After the mobile phone receives the target image frame data, it can display two pictures, one is the real-time stream, and the other is the video picture after the target tracking object is zoomed. For details, please refer to Fig. 9 . When displayed on a mobile phone, the mobile phone decodes the image data frame to obtain the interactive touch elements of all objects to be selected in the video image (that is, the rectangular frame that selects the objects to be selected). The user touches the interactive touch elements according to actual needs, and the mobile phone sends the identity information of the object to be selected corresponding to the touched interactive touch elements to the camera as the target identity information. The camera re-determines the target tracking object based on the received target identity information and tracks it, and sends the target image frame data of the target tracking object to the mobile phone.

[0098] After that, the mobile phone parses the received adjacent first target image frame data and second target image frame data to obtain the first position parameter and the second position parameter of the target tracking object. That is, the coordinates of the two diagonal vertices A1 and B1 of the first video image are (XA1, YA1) and (XB1, YB1), and the coordinates of the two diagonal vertices A2 and B2 of the second video image are (XA2, YA2) and (XB2, YB2). When the target refresh rate is 60fps and the encoding frame rate is 15 frames, the step size is determined by combining the first position parameter and the second position parameter. Accordingly, the canvas step size can be set to ((XA2-XA1) / 4, (YA2-YA1) / 4), and ((XB2-XB1) / 4, (YB2-YB1) / 4).

[0099] After the mobile phone receives the first image frame data, it is passed to the hardware decoder for rendering. The mobile phone player notifies the upper UI to start rendering the first image frame data. At time t = 0.0ms, the mobile phone screen displays the first image; at time t = 16.66ms, since the player has no data to render at this time, the screen still displays the first video image corresponding to the first image frame data, and the mobile phone will adjust the canvas by one canvas step; at time t = 16.66ms + 16.66ms, since the player has no data to render at this time, the screen still displays the first video image corresponding to the first image frame data, and the mobile phone will continue to adjust the canvas by one canvas step.

[0100] At time t=16.66ms+16.66ms+16.66ms, the player also has no data to render, and the screen still displays the first video image corresponding to the first image frame data. The mobile phone will continue to adjust the canvas by one canvas step. At time t=16.66ms+16.66ms+16.66ms+16.66ms, since the player starts rendering the second image frame data, the mobile phone will adjust the canvas to display the second video image rendered by the second target image frame data.

[0101] After completing the video image interpolation between the first target image frame data and the second target image frame data, the third target image frame data is received, and the same method is used to complete the interpolation processing between the second target image frame data and the third target image frame data. This is continuously executed to achieve a 60fps refresh rate screen display, thereby solving the problem of video image freeze of the target tracking object during the zooming process.

[0102] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a monitoring image tracking device for implementing the monitoring image tracking method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more monitoring image tracking device embodiments provided below can refer to the limitations of the monitoring image tracking method above, and will not be repeated here.

[0104] See also Fig.10 , a monitoring image tracking device includes: an image frame receiving module 1002, an image frame parsing module 1004, an interactive response module 1006 and a target image frame receiving module 1008.

[0105] The image frame receiving module 1002 is used to receive image frame data collected and sent by the image collector; the image frame parsing module 1004 is used to parse and display the interactive contacts corresponding to each selected object in the image frame data; the interactive response module 1006 is used to respond to the interactive actions of the interactive contacts, and send the target identity information of the selected object corresponding to the interactive contacts to the image collector; the target image frame receiving module 1008 is used to receive the target image frame data tracked and sent by the image collector.

[0106] See also Fig.11 In one embodiment, after the target image frame receiving module 1008 , the device further includes a frame insertion control module 112 .

[0107] The interpolation control module 112 is used to determine the canvas step length between adjacent video images to be displayed based on adjacent first target image frame data and second target image frame data; based on the canvas step length, a differential video image is inserted and displayed between the first video image rendered by the first target image frame data and the second video image rendered by the second target image frame data.

[0108] In one embodiment, the interpolation control module 112 is also used to parse adjacent first target image frame data and second target image frame data to determine first position parameters and second position parameters of the target tracking object; and determine the canvas step length between adjacent video images to be displayed based on the first position parameters and the second position parameters.

[0109] In one embodiment, the interpolation control module 112 is also used to obtain the target refresh rate and the encoding frame rate of the image collector; determine the number of required differentiated video images based on the encoding frame rate and the target refresh rate; and determine the canvas step size between adjacent video images to be displayed based on the number, the first position parameter and the second position parameter.

[0110] In one embodiment, the interpolation control module 112 is also used to determine a preset refresh time interval based on the encoding frame rate of the image collector; display a first video image rendered by the first target image frame data at the initial moment; each time the preset refresh time interval is increased, the canvas is adjusted according to the canvas step size, and the first video image is displayed as a differential video image; if the inserted differential video images reach the required number, the second video image rendered by the second target image frame data is displayed.

[0111] See also Fig.12 , a monitoring image tracking device includes: an image frame acquisition module 122, a target identity receiving module 124 and a target tracking module 126.

[0112] The image frame acquisition module 122 is used to acquire image frame data and send it to the terminal device; the target identity receiving module 124 is used to receive the target identity information sent by the terminal device and determine the target tracking object; the target tracking module 126 is used to track and acquire the target image frame data of the target tracking object and send it to the terminal device.

[0113] In one embodiment, the image frame acquisition module 122 is further configured to track the candidate objects in the image frame data according to a preset priority order.

[0114] In one embodiment, the image frame acquisition module 122 is further used to obtain video code stream data and determine objects to be selected; configure identity information and location information for each object to be selected to obtain image frame data.

[0115] Each module in the above-mentioned monitoring image tracking device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the terminal device in the form of hardware, or can be stored in the memory in the terminal device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0116] The above-mentioned monitoring image tracking device, after the image collector acquires the image frame data through image acquisition, transmits it to the terminal device for parsing, obtains and displays the interactive touches corresponding to each selected object in the image frame data. The user only needs to touch the interactive touches and send the target identity information of the selected object corresponding to the interactive touches to the image collector, so that the selected object can be determined as the target tracking object, track it, and return the target image data frame corresponding to the target tracking object to the terminal device. Through this solution, the user can freely select and switch the target tracking object on the terminal device, thereby actively selecting or changing the tracking target of the image collector.

[0117] In one embodiment, a terminal device is provided, whose internal structure diagram can be as follows: Fig.13As shown. The terminal device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the terminal device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a monitoring image tracking method is implemented. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the terminal device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the terminal device shell, or an external keyboard, touchpad or mouse, etc.

[0118] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] In one embodiment, a terminal device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0120] Receive image frame data collected and sent by the image collector;

[0121] Parsing and displaying the interactive touch elements corresponding to each selected object in the image frame data;

[0122] In response to the interactive action of the interactive touch element, the target identity information of the selected object corresponding to the interactive touch element is sent to the image collector;

[0123] The receiving image collector tracks the target tracking object and sends the target image frame data.

[0124] In one embodiment, an image collector is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0125] Collect image frame data and send it to the terminal device;

[0126] Receive the target identity information sent by the terminal device and determine the target tracking object;

[0127] The target tracking object is tracked and collected to obtain the target image frame data, and sent to the terminal device.

[0128] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0129] Receive image frame data collected and sent by the image collector;

[0130] Parsing and displaying the interactive touch elements corresponding to each selected object in the image frame data;

[0131] In response to the interactive action of the interactive touch element, the target identity information of the selected object corresponding to the interactive touch element is sent to the image collector;

[0132] The receiving image collector tracks the target tracking object and sends the target image frame data.

[0133] Or, when the computer program is executed by a processor, the following steps are implemented:

[0134] Collect image frame data and send it to the terminal device;

[0135] Receive the target identity information sent by the terminal device and determine the target tracking object;

[0136] The target tracking object is tracked and collected to obtain the target image frame data, and sent to the terminal device.

[0137] The above-mentioned terminal device, image collector and computer storage medium, after the image collector performs image acquisition to obtain image frame data, transmit it to the terminal device for parsing, obtain and display the interactive touch elements corresponding to each selected object in the image frame data. The user only needs to touch the interactive touch element and send the target identity information of the selected object corresponding to the interactive touch element to the image collector, so that the selected object can be determined as the target tracking object, track it, and return the target image data frame corresponding to the target tracking object to the terminal device. Through this solution, the user can freely select and switch the target tracking object on the terminal device, thereby actively selecting or changing the tracking target of the image collector.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A monitoring image tracking method, characterized in that: include: Receive image frame data collected and sent by the image collector; Parsing and displaying the interactive touch items corresponding to each to-be-selected object in the image frame data; The interactive touch element is associated with the identity information of the object to be selected; In response to the interactive action of the interactive touch element, the target identity information of the to-be-selected object corresponding to the interactive touch element is sent to the image collector; The target identity information is used to determine the target tracking object; Receive target image frame data sent by the image collector after tracking the target tracking object.

2. The monitoring image tracking method according to claim 1, characterized in that: After receiving the target image frame data sent by the image collector tracking the target tracking object, the method further includes: Determine a canvas step length between adjacent video images to be displayed according to adjacent first target image frame data and second target image frame data; According to the canvas step, a differential video image is inserted and displayed between a first video image rendered by the first target image frame data and a second video image rendered by the second target image frame data; the differential video image is determined according to the first video image.

3. The monitoring image tracking method according to claim 2, characterized in that: The step of determining the canvas step length between adjacent video images to be displayed according to adjacent first target image frame data and second target image frame data includes: Parsing adjacent first target image frame data and second target image frame data to determine first position parameters and second position parameters of the target tracking object; A canvas step length between adjacent video images to be displayed is determined according to the first position parameter and the second position parameter.

4. The monitoring image tracking method according to claim 3, characterized in that: The step of determining the canvas step length between adjacent video images to be displayed according to the first position parameter and the second position parameter includes: Obtaining a target refresh rate and an encoding frame rate of the image collector; Determining the number of required differential video images according to the encoding frame rate and the target refresh rate; A canvas step length between adjacent video images to be displayed is determined according to the quantity, the first position parameter, and the second position parameter.

5. The monitoring image tracking method according to any one of claims 2 to 4, characterized in that: The canvas step lengths between adjacent video images to be displayed are all consistent.

6. The monitoring image tracking method according to claim 5, characterized in that: The inserting of a differential video image between a first video image rendered by the first target image frame data and a second video image rendered by the second target image frame data according to the canvas step size comprises: Determining a preset refresh time interval according to the encoding frame rate of the image collector; Displaying a first video image rendered by the first target image frame data at an initial moment; Each time the preset refresh time interval is increased, after adjusting the canvas according to the canvas step length, the first video image is displayed as a differential video image; If the inserted differential video images reach the required number, the second video image rendered by the second target image frame data is displayed.

7. The monitoring image tracking method according to claim 6, characterized in that: Also includes: If, during the process of inserting the differential video image, the third target image frame data is received, the differential video image insertion between the first video image and the second video image is terminated; re-determining the canvas step length according to the second target image frame data and the third target image frame data; Inserting and displaying a differential video image between the second video image and a third video image rendered by the third target image frame data according to the re-determined canvas step size; A differential video image is determined based on the second video image.

8. The monitoring image tracking method according to any one of claims 2 to 4, characterized in that: The canvas step sizes between adjacent video images to be displayed are not exactly the same.

9. The monitoring image tracking method according to any one of claims 1 to 4, characterized in that: Track the target tracking object, including scaling the target tracking object or outputting the position information of the target tracking object.

10. A monitoring image tracking method, characterized in that: include: Collect image frame data and send it to the terminal device; The image frame data is used by the terminal device to parse and obtain the interactive touch elements corresponding to each object to be selected; The interactive touch element is associated with the identity information of the object to be selected; receiving target identity information sent by the terminal device, and determining a target tracking object; the target identity information is sent by the terminal device in response to an interactive action of the interactive contact; The target tracking object is tracked and collected to obtain target image frame data, and sent to the terminal device.

11. The monitoring image tracking method according to claim 10, characterized in that: After collecting image frame data, it also includes: Target tracking is performed on the objects to be selected in the image frame data according to a preset priority order.

12. The monitoring image tracking method according to claim 10, characterized in that: The collecting of image frame data comprises: Obtain video code stream data and determine the object to be selected; Identity information and location information are configured for each of the objects to be selected to obtain image frame data.

13. A monitoring image tracking device, characterized in that: include: An image frame receiving module is used to receive image frame data collected and sent by an image collector; An image frame parsing module, used for parsing and displaying the interactive touch corresponding to each object to be selected in the image frame data; the interactive touch is associated with the identity information of the object to be selected; An interactive response module, used to respond to the interactive action of the interactive touch element, and send the target identity information of the to-be-selected object corresponding to the interactive touch element to the image collector; The target identity information is used to determine the target tracking object; The target image frame receiving module is used to receive the target image frame data sent by the image collector when tracking the target tracking object.

14. A monitoring image tracking device, characterized in that: include: An image frame acquisition module, used to acquire image frame data and send it to a terminal device; The image frame data is used by the terminal device to parse and obtain the interactive touch corresponding to each object to be selected; the interactive touch is associated with the identity information of the object to be selected; A target identity receiving module, used to receive target identity information sent by the terminal device and determine a target tracking object; the target identity information is sent by the terminal device in response to an interactive action of the interactive contact; The target tracking module is used to track and collect the target tracking object to obtain target image frame data, and send it to the terminal device.

15. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the monitoring image tracking method according to any one of claims 1 to 9 are implemented.

16. An image collector, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the monitoring image tracking method according to any one of claims 10 to 12 are implemented.

Images