Image processing method, device and processing equipment

Abstract

The present application is applicable to the field of monitoring technology, and provides an image processing method, apparatus and processing equipment, including: obtaining image features of each image frame in a video to be processed, wherein the video to be processed is obtained by shooting with a camera; determining an image frame matching the reference image according to the image features of each image frame and the image features of a reference image, and obtaining a target image frame, wherein the reference image is an image frame shot by the camera in a non-tracking state; according to the position of the target image frame in the video to be processed, filtering out tracking image frames shot by the camera in a tracking state from the video to be processed, and / or filtering out non-tracking image frames shot by the camera in a non-tracking state from the video to be processed. Through the above method, the speed at which users obtain effective information can be improved.

Description

Technical Field

[0001] The present application belongs to the field of monitoring technology, and in particular, relates to an image processing method, apparatus, processing device, and computer-readable storage medium. Background Art

[0002] Surveillance cameras are an important part of security systems. They can provide accurate, real-time, intuitive and rich monitoring information.

[0003] With the development of information technology and the improvement of people's living standards, people's awareness of safety precautions is also increasing. More and more surveillance cameras are installed in urban streets, squares, parks, stations and / or residential buildings to monitor these places. Relevant departments can not only identify abnormal situations in the monitored areas in a timely manner based on the monitoring information obtained from the surveillance cameras, but also better ensure people's safety.

[0004] Since the monitoring information is output in the form of video, and the information in the video is relatively complex, it is difficult for users to quickly obtain effective information from the video. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, an apparatus, and a processing device, which can solve the problem that it is difficult for users to quickly obtain effective information from videos.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Obtaining image features of each image frame to be matched in a video to be processed, wherein the video to be processed is obtained by being shot by a camera;

[0008] According to the image features of each of the image frames to be matched and the image features of the reference image, an image frame matching the reference image is determined to obtain a target image frame, wherein the reference image is an image frame captured by the camera in a non-tracking state;

[0009] According to the position of the target image frame in the video to be processed, the tracking image frames captured by the camera in the tracking state are filtered out from the video to be processed, and / or the non-tracking image frames captured by the camera in the non-tracking state are filtered out from the video to be processed.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0011] An image feature acquisition module is used to acquire image features of each image frame to be matched in a video to be processed, where the video to be processed is captured by a camera;

[0012] a target image frame determination module, configured to determine an image frame that matches the reference image according to image features of the image frames to be matched and image features of a reference image, and obtain a target image frame, wherein the reference image is an image frame captured by the camera in a non-tracking state;

[0013] A tracking image frame filtering module is used to filter out tracking image frames taken by the camera in a tracking state from the video to be processed according to the position of the target image frame in the video to be processed, and / or to filter out non-tracking image frames taken by the camera in a non-tracking state from the video to be processed.

[0014] In a third aspect, an embodiment of the present application provides a processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a processing device, enables the processing device to execute the method described in the first aspect above.

[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0018] In the embodiment of the present application, the image features of each image frame to be matched in the video to be processed and the image features of the reference image are obtained to determine the image frame matching the reference image, and obtain the target image frame. Since the reference image is the image frame captured by the camera in a non-tracking state, the image frame matching the reference image (i.e., the target image frame) is the image frame captured by the camera in a non-tracking state. Therefore, according to the position of the target image frame in the video to be processed, the tracking image frame captured by the camera in a tracking state can be filtered out from the video to be processed, and / or the non-tracking image frame captured by the camera in a non-tracking state can be filtered out from the video to be processed. Since the tracking image frame and / or the non-tracking image frame can be filtered out from the video to be processed, that is, by classifying the image frames in the video to be processed, it is beneficial for the user to quickly obtain effective information based on the filtered tracking image frame and / or the non-tracking image frame, thereby improving the speed at which the user obtains effective information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art.

[0020] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of matching points of two image frames when a gimbal is in a reset position provided by an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of matching points of two image frames when a gimbal is in a non-reset position provided by an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of the effect after a progress bar of a video to be processed is preliminarily marked according to an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of the effect after the preliminary identification of the progress bar of a video to be processed is corrected, provided by another embodiment of the present application;

[0025] Figure 6 This is a schematic diagram of an effect in which a progress bar of a video to be processed is marked as three types of progress bar intervals provided by an embodiment of the present application;

[0026] Figure 7 is a flowchart of another image processing method provided by another embodiment of the present application;

[0027] Figure 8 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0028] Fig. 9 It is a schematic diagram of the structure of the processing device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0030] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0031] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0034] Embodiment 1:

[0035] After installing a surveillance camera in an area that needs to be monitored, users can obtain relevant videos from the surveillance camera.

[0036] For example, if the installed surveillance camera is a common surveillance camera, the maximum visual range of the obtained video is usually fixed. Since the video of the picture beyond the maximum visual range cannot be obtained, the information of interest to the user may be lost.

[0037] If the installed surveillance camera is a surveillance camera that can track targets, the user only needs to configure the area of ​​interest and the target. In this way, after the target tracking function is triggered, the surveillance camera can continuously track the target by changing the visual range (such as by rotating the pan / tilt to change the visual range).

[0038] However, since the video output by a surveillance camera that can track a target includes both tracking image frames taken by the surveillance camera in a tracking state and non-tracking image frames taken by the surveillance camera in a non-tracking state, and the tracking image frames and the non-tracking image frames are not effectively distinguished, it is difficult for users to quickly obtain effective information from the video.

[0039] In order to facilitate users to quickly obtain effective information from a video, an embodiment of the present application provides an image processing method. In the image processing method, a target image frame is found by matching the image features of an image frame in the video to be processed with the image features of a reference image (the target image frame is an image frame in the video to be processed that matches the reference image), and the video to be processed is classified into a tracking image frame and a non-tracking image frame according to the target image frame.

[0040] The image processing method provided in the embodiments of the present application is described below with reference to the accompanying drawings.

[0041] Figure 1 A flowchart of an image processing method provided by an embodiment of the present application is shown, which is described in detail as follows:

[0042] Step S11, obtaining image features of each image frame to be matched in a video to be processed, where the video to be processed is captured by a camera.

[0043] The above-mentioned image features may be global features of the image frame or local features of the image frame.

[0044] The visual range of the camera can be changed, for example, the visual range of the camera can be changed by rotating the pan / tilt.

[0045] In some embodiments, the video captured by a camera whose visual range can be switched by rotating the pan / tilt head is used as the video to be processed.

[0046] The video to be processed includes multiple image frames, and each image frame generally includes multiple image features. The number of image features included in an image frame is related to the texture richness of the image frame itself. If the texture of the image frame itself is relatively rich, the number of image features included in the image frame is also relatively large. Conversely, the number of image features included in the image frame is also relatively small. Since the image frames of the video to be processed generally need to perform subsequent matching actions, for the sake of ease of description, the image frames in the video to be processed are subsequently referred to as image frames to be matched.

[0047] In an embodiment of the present application, the image features of the image frame to be matched can be extracted through a preset network model, or the image features of the image frame to be matched can be extracted through a preset algorithm, such as extracting local texture features of the image through a local binary pattern (Local Binary Patterns, LBP) algorithm.

[0048] Step S12, determining an image frame matching the reference image according to the image features of the image frames to be matched and the image features of the reference image, and obtaining a target image frame, wherein the reference image is an image frame captured by the camera in a non-tracking state.

[0049] In some embodiments, an image frame captured by a predetermined camera in a non-tracking state is used as a reference image. For example, assuming that the visual range of the camera is adjusted by the pan / tilt, when the pan / tilt is in position A (i.e., the position where the pan / tilt has performed a reset action and has not yet rotated again, i.e., the pan / tilt is in the reset position), the field of view of the camera on the pan / tilt is the same, i.e., the image frames captured by the camera are usually the same (in the same field of view, the objects that cannot be moved are the same, and the objects that can be moved are usually different), i.e., the camera is in a non-tracking state; conversely, when the pan / tilt is not in position A, since the field of view of the camera changes, it indicates that the camera on the pan / tilt is in a tracking state. Therefore, in the embodiments of the present application, an image frame captured by the camera when the pan / tilt is in position A can be used as a reference image.

[0050] In some embodiments, considering that the first image frame of the video to be processed is usually an image frame captured by the camera when the pan / tilt head is in the reset position, the first image frame of the video to be processed can be used as a reference image, and the remaining image frames of the video to be processed can be used as image frames that need to be matched with the reference image.

[0051] After obtaining the reference image, the image features corresponding to the reference image are obtained. The image features of the reference image can be obtained by using the same method as that of obtaining the image features of each image frame to be matched in the video to be processed, which will not be described in detail here.

[0052] In the embodiment of the present application, each image frame to be matched in the video to be processed is matched with the reference image respectively, so as to search for an image frame matching the reference image from each image frame to be matched. Specifically, for each image frame to be matched in the video to be processed, each image feature in the image frame to be matched is matched with each image feature of the reference image respectively, so as to search for each image feature matching each image feature of the reference image in the image frame to be matched, and if the number of image features matching each image feature of the reference image found in one image frame to be matched is greater than a preset threshold, then it is determined that the image frame to be matched matches the reference image.

[0053] In some embodiments, when the video to be processed is shot by a camera with a pan-tilt rotation, the position of the pan-tilt will not change or will only change slightly after the pan-tilt has performed a reset action and before it rotates again. Therefore, after the pan-tilt has performed a reset action and before it rotates again, the pixel coordinates of the feature point pairs matched in the reference image and the image frame to be matched are consistent or have only a slight offset. That is, in this embodiment, the pixel coordinates of the feature point pairs matched in the target image frame and the reference image are consistent or have only a slight offset. That is, in this embodiment of the present application, if the reference image is an image frame shot by the camera when the pan-tilt is in the reset position, whether the pan-tilt is in the reset position can be determined based on the matching result of whether the image frame to be matched matches the reference image.

[0054] like Figure 2 As shown in FIG, it is assumed that image frame 1 is the reference image, image frame 2 is the image frame to be matched, and the feature points of the line connecting image frame 1 and image frame 2 are the matching feature point pairs. Figure 2 It can be seen that the pixel coordinates of the two matched feature points in image frame 1 and image frame 2 are the same, which indicates that image frame 2 is also an image frame captured by the camera when the gimbal is in the reset position (that is, the gimbal has performed the reset action and has not rotated again). In some embodiments, it can be determined in the following way whether the feature points of the two image frames are feature points captured by the camera when the gimbal is in the reset position: assuming that the i-th image feature Fbi in the reference image matches the j-th image feature Fmj in the image frame to be matched, and let the pixel coordinates of Fbi be {Xbi, Ybi}, the pixel coordinates of Fmj be {Xmj, Ymj}, and the allowable offset of the reset position matching feature point is not greater than P. If

[0055] |Xbi–Xmj|<=P;

[0056] |Ybi–Ymj|<=P;

[0057] That is, if the offset between Fbi and Fmj is less than or equal to P, it is considered that the feature point Fmj of the image frame to be matched is a feature point captured by the camera when the pan / tilt head is in the reset position.

[0058] The number of feature points in the image frame to be matched is counted, and if the number is greater than or equal to the preset number threshold J, it is determined that the pan / tilt is in the reset position at this time, and the image frame to be matched is the image frame captured by the camera when the pan / tilt is in the reset position (the image frame to be matched is also the image frame captured by the camera in the non-tracking state). That is, before determining whether the pan / tilt is in the reset position, the allowable offset P and the number threshold J are preset, and then P and J are combined to determine whether the pan / tilt is in the reset position.

[0059] like Figure 3 As shown, when there is a matching pair of feature points, but the pixel coordinates of the two matching feature points in the image frame are inconsistent, or there is no matching pair of feature points, it indicates that image frame 2 is an image frame taken by the camera when the gimbal is in a non-reset position (that is, the gimbal is rotating), that is, it indicates that image frame 2 is a tracking image frame taken by the camera in a tracking state.

[0060] In some embodiments, after matching a to-be-matched image frame with a reference image, feature point pairs that are incorrectly matched are eliminated. For example, feature point pairs that are incorrectly matched are eliminated by random sample consensus (RANSAC). Specifically, the feature points in the reference image are used to find the most similar feature points in the to-be-matched image frame according to the k-nearest neighbor algorithm to form a matching pair. The following processing is then performed: 1) Randomly select any 4 matching feature point pairs. 2) Calculate the homography matrix based on these 4 matching feature point pairs, and use the error metric function to calculate the error between other matching feature point pairs and the homography matrix. 3) Preset an error threshold U. If the calculated error is greater than U, add the matching feature point pair to the external set, otherwise add it to the internal set and count the total matching error. 4) If the total error is greater than the preset total error threshold, repeat steps 1) to 4), otherwise select the internal set with the largest number of matching feature point pairs as the optimal solution. Keep the matching feature point pairs in the internal set corresponding to the optimal solution, and eliminate the matching feature point pairs in the external set. Since the number of mismatched feature point pairs is limited, the impact of mismatches on tracking state recognition is reduced, thereby improving the accuracy of tracking state recognition.

[0061] Step S13, according to the position of the above-mentioned target image frame in the above-mentioned video to be processed, filtering out the tracking image frames taken by the above-mentioned camera in the tracking state from the above-mentioned video to be processed, and / or filtering out the non-tracking image frames taken by the above-mentioned camera in the non-tracking state from the above-mentioned video to be processed.

[0062] In the embodiment of the present application, all the matched target image frames are determined to be non-tracking image frames obtained by the camera in a non-tracking state, and each image frame in the video to be processed that is not a target image frame is determined to be a tracking image frame obtained by the camera in a tracking state. For example, assuming that the video to be processed includes 60 image frames, and only the first image frame and the 60th image frame are determined to be non-tracking image frames, then each image frame between the first image frame and the 60th image frame is determined to be a tracking image frame.

[0063] In an embodiment of the present application, an image frame matching the reference image is determined by acquiring the image features of each image frame in the video to be processed and the image features of the reference image, and a target image frame is obtained. Since the reference image is an image frame captured by the camera in a non-tracking state, the image frame matching the reference image (i.e., the target image frame) is an image frame captured by the camera in a non-tracking state. Therefore, according to the position of the target image frame in the video to be processed, it is possible to filter out the tracking image frames captured by the camera in a tracking state from the video to be processed, and / or, filter out the non-tracking image frames captured by the camera in a non-tracking state from the video to be processed. Since the tracking image frames and / or the non-tracking image frames are filtered out from the video to be processed, that is, by classifying the image frames in the video to be processed, it is beneficial for the user to quickly obtain effective information based on the filtered tracking image frames and / or non-tracking image frames, thereby improving the speed at which the user obtains effective information.

[0064] In some embodiments, after the above step S13, the method further includes:

[0065] A1. According to the tracking image frame, a progress bar interval corresponding to the tracking image frame is determined on the progress bar of the video to be processed to obtain a first type of progress bar interval. And\or, according to the non-tracking image frame, a progress bar interval corresponding to the non-tracking image frame is determined on the progress bar of the video to be processed to obtain a second type of progress bar interval.

[0066] A2. Marking the first type of progress bar interval and / or the second type of progress bar interval on the progress bar of the video to be processed.

[0067] In an embodiment of the present application, the progress bar intervals corresponding to the tracking image frames and the non-tracking image frames are determined based on the total number of image frames included in the video to be processed, the length of the progress bar of the video to be processed, and the sorting relationship of the target image frames in the video to be processed.

[0068] It should be pointed out that, since a video to be processed may include image frames captured by the camera in multiple tracking states and multiple non-tracking states (such as when the pan / tilt is in the reset position, the camera is in the non-tracking state, and when the pan / tilt is in the non-reset position, the camera is in the tracking state), the image frames captured by the camera in the tracking state may include image frames captured by the camera in the tracking state in different time periods, so that the first type of progress bar interval subsequently determined based on the tracking image frames may also include multiple progress bar intervals, that is, in an embodiment of the present application, multiple first type of progress bar intervals may be determined on the progress bar of the video to be processed, that is, on the progress bar of the video to be processed, the number of first type of progress bar intervals is usually greater than 1.

[0069] In the embodiment of the present application, a method similar to the method for determining the first type of progress bar interval may be used to determine the second type of progress bar interval. Alternatively, the progress bar interval on the progress bar of the video to be processed that is not the first type is directly determined as the second type of progress bar interval. It should be noted that the number of the second type of progress bar intervals may be greater than 1.

[0070] In an embodiment of the present application, different identifiers may be used to identify the first type of progress bar interval and the second type of progress bar interval on the progress bar of the video to be processed, respectively. Among them, the identification may be performed by information such as color and / or size that is convenient for users to distinguish. For example, the first type of progress bar interval on the progress bar of the video to be processed is filled with green, and the second type of progress bar interval on the progress bar of the video to be processed is filled with gray. Of course, considering that there are only two types of progress bar intervals, only one type of progress bar interval may be identified. For example, a color different from the original filling color of the progress bar of the video to be processed is determined as the target color, and then the target color is used to fill the second type of progress bar interval on the progress bar of the video to be processed, while the filling color of the first type of progress bar interval continues to remain the same as the original filling color of the progress bar of the video to be processed.

[0071] In an embodiment of the present application, after the first type of progress bar interval and / or the above-mentioned second type of progress bar interval are marked on the progress bar of the video to be processed, the user can quickly determine, based on the mark of the progress bar, which image frame corresponding to which progress bar interval is the tracking image frame taken by the camera in the tracking state, and / or quickly determine which image frame corresponding to which progress bar interval is the non-tracking image frame taken by the camera in the non-tracking state, thereby effectively improving the speed at which the user obtains effective information from the video.

[0072] In some embodiments, after the above step A1, the method further includes:

[0073] B1. Determine a progress bar interval to be corrected, where the progress bar interval to be corrected includes: a first type of progress bar interval whose corresponding video duration is less than a preset duration threshold.

[0074] The inventors of the present application have analyzed and found that since there is a minimum duration required for the target tracking to trigger and reset, when the continuous duration (i.e., video duration) of a tracking state is determined to be less than C, it is determined that mis-identification has occurred in the image frame sequence corresponding to the tracking state (i.e., non-tracking image frame sequence). For example, assume that the minimum duration is C (i.e., the preset duration threshold is C), and the video duration corresponding to the progress bar interval M of the first type is T. If T < C, it is determined that mis-identification has occurred in the type of the progress bar interval corresponding to M, that is, M is a progress bar interval to be corrected.

[0075] B2. Correct the above progress bar interval to be corrected from the progress bar interval of the first type to the progress bar interval of the second type.

[0076] In the embodiment of the present application, since the duration from the trigger to the reset of the target tracking is not shorter than the preset duration threshold, correcting the progress bar interval of the first type with a video duration less than the preset duration threshold to the progress bar interval of the second type can improve the accuracy of the type to which the finally obtained progress bar interval belongs. For example, assume that after the progress bar of the video to be processed is preliminarily identified as Figure 4 shown, the playing duration of the progress bar of the video to be processed is 10 minutes. The progress bar interval filled with white is the progress bar interval of the first type, and the image frames corresponding to the progress bar interval of the first type are the image frames obtained when the camera is in the tracking state (i.e., the pan-tilt is in the non-reset state). The progress bar interval filled with black is the progress bar interval of the second type. Assume that the playing durations of the progress bar intervals a and b are less than the preset duration threshold, then the progress bar intervals a and b are the progress bar intervals to be corrected in the embodiment of the present application and will be corrected to the progress bar intervals of the second type. The progress bar of the video to be processed after correction is as Figure 5 shown.

[0077] Of course, it is also possible to determine whether the progress bar interval is a progress bar interval to be corrected by counting the number of image frames included in the progress bar interval. Specifically, define a minimum tracking frame number threshold of size Win (Win = C * video frame rate), and count the number of image frames (the number of tracking image frames) in each tracking time period of the video (i.e., count each progress bar interval of the first type of the video). If the tracking frame number is less than Win, set the progress bar interval corresponding to this time period to the progress bar interval of the second type, and correct the image frames of this time period to the non-tracking image frames obtained when the camera is in the non-tracking state (i.e., the pan-tilt is in the reset state), otherwise retain the current state (i.e., the progress bar interval to be corrected is still identified as the progress bar interval of the first type).

[0078] In some embodiments, considering that the user is more concerned about the target tracked by the camera, if it is detected that the function of tracking the video is selected, the tracking image frame in the video to be processed is directly played after receiving the video playback instruction subsequently, wherein, when the function of tracking the video is selected, the tracking image frame will be directly played without playing the non-tracking image frame. Specifically, the selection item corresponding to the function of tracking the video is provided on the display interface of the player, and when the user selects the selection item, it indicates that the user wants to play only the tracking image frame. After the user clicks the play button, the player will receive the video playback instruction and play only the tracking image frame in the video to be processed.

[0079] In some embodiments, considering that the background objects that can be photographed by the surveillance camera are usually the same, while the foreground objects (such as pedestrians, vehicles, etc.) are usually different, and the foreground objects will also interfere with the calculation of the similarity of the background objects, therefore, in order to improve the accuracy and speed of background object matching, before the above step S12, it also includes:

[0080] Invalid image features of each image frame to be matched are eliminated, wherein the image features located at the position of the foreground object of the image frame to be matched are invalid image features of the image frame to be matched.

[0081] Correspondingly, the above step S12 includes:

[0082] An image frame matching the reference image is determined based on the image features of the image frames from which invalid image features have been eliminated and the image features of the reference image.

[0083] Specifically, the following method is used to eliminate invalid image features of the image frame to be matched: the target detection method is used to obtain the foreground object position (bounding box) and image feature set of the image frame to be matched (each image feature set is used to store each image feature of the corresponding image frame to be matched) ， And remove the image features whose positions intersect with the foreground object positions. For example, suppose the foreground object set is B, that is, a single boundingbox b1,b2…bi…∈B, let the image feature set be T, record the position of a single image feature as ti, that is, t1,t2…ti…∈T, the image feature set to be retained is S1, and the image feature set to be removed is D, then:

[0084] D = {ti|if ti∈B};

[0085]

[0086] In the embodiment of the present application, since the foreground object will interfere with the calculation of the similarity of the background object, treating the irrelevant motion foreground features as invalid image features can significantly improve the accuracy of subsequent background matching.

[0087] In some embodiments, in order to further improve the accuracy of matching, in addition to removing invalid image features of the image frame to be matched, invalid image features in the reference image are also removed, and then the image frame to be matched from which the invalid image features are removed is matched with the reference image from which the invalid image features are removed, which will not be described in detail here. Among them, the image features located at the position of the foreground object of the reference image frame are the invalid image features of the above-mentioned reference image frame, and the method of removing invalid image features in the reference image is the same as that of removing invalid image features of the image frame to be matched, which will not be described in detail here.

[0088] In some embodiments, the above-mentioned image features include feature positions and feature confidences. Since the feature confidences will affect the success rate of matching, and since the matching process is based on the image features of the reference image, that is, the image features most similar to the image features of the reference image are searched from the image feature set of the image frame to be matched, the amount of calculation required for matching can be effectively reduced by thinning the reference image before performing subsequent matching actions. At this time, before the above-mentioned step S12, it also includes:

[0089] C1. Determine the target area according to the above feature positions and the preset area radius.

[0090] C2. If the number of the image features of the reference image in the target area is less than a preset number threshold, retaining all the image features of the reference image in the target area.

[0091] C3. If the number of image features of the above-mentioned benchmark images in the above-mentioned target area is not less than the above-mentioned quantity threshold, the image features of the above-mentioned benchmark images in the above-mentioned target area are sorted in descending order according to the feature confidence of the above-mentioned image features, and the image features of the above-mentioned benchmark images with the top N sorting are retained, wherein the above-mentioned N is equal to the above-mentioned quantity threshold.

[0092] Correspondingly, the above step S12 includes:

[0093] An image frame matching the reference image is determined based on the image features of each of the image frames and the image features retained in the reference image.

[0094] In the embodiment of the present application, the number of image features within a preset area radius of the feature points of the reference image (i.e., the pixel points where the image features are obtained) is counted, and the image features are sorted in descending order according to the feature confidence, and finally the image features outside the number threshold are eliminated to obtain the retained image features. Specifically, 1) a feature point of the reference image is selected as a sparse feature point, and a circular area with a radius length of H is selected with the sparse feature point as the center, and the image features appearing in the circular area are counted to obtain a set S2, and each image feature in the set S2 is sorted in descending order according to the corresponding feature confidence; 2) a specific feature quantity threshold R is set. If the number of image features in the set S2 is greater than R, only the first R image features are retained (i.e., the first R image features are the screened and retained image features), otherwise all image features in the set S2 are retained.

[0095] In the embodiment of the present application, after obtaining the image features retained in the reference image, the image features retained in the reference image are matched with the image features of the image to be matched to determine whether the image frame to be matched is a tracking image frame. When the texture of the reference image (such as the texture of a local area) is relatively rich, the number of image features detected in the area is also relatively large, and not all image features can be effectively matched (for example, image features with high feature confidence often have better matching effects, while image features with low feature confidence are prone to mismatching). Moreover, when there are more image features, more computing resources are consumed, the required computing waiting time is longer, and the memory usage is larger. Therefore, by sparsely treating the reference image, the image features with low feature confidence can be effectively reduced, thereby effectively reducing the amount of calculation required in the subsequent matching process, thereby improving the accuracy and speed of subsequent matching.

[0096] In some embodiments, the image feature also includes a feature vector, and the image feature is represented by the feature vector. That is, in the matching process, whether two image features corresponding to the two feature vectors match is determined by comparing whether the two feature vectors match.

[0097] In some embodiments, the reference image may be thinned, and invalid image features of the reference image and the image frame to be matched (or invalid features of the image frame to be matched) may be eliminated before executing step S12. That is, before executing step S12, the above steps C1 to C3 are first executed, and the above step of "eliminating invalid image features of each image frame to be matched" is first executed. Since matching is performed based on the screened retained image features, it is not affected by foreground objects, and since the higher the feature confidence, the higher the matching success rate, therefore, by eliminating invalid image features and image features screened based on feature confidence, the subsequent matching speed can be faster and more accurate.

[0098] In some embodiments, considering that when the camera is in a tracking state (i.e., the pan-tilt is not in the reset position), the camera may be in motion or in a stationary state. For example, if the pan-tilt is in motion, the camera on the pan-tilt is also in motion, which indicates that the camera is tracking the target. If the pan-tilt is in a stationary state, the camera on the pan-tilt is also in a stationary state, which indicates that the camera has lost the tracking target and has stopped tracking the target. Therefore, in order to make a more detailed division of the tracking image frames captured by the camera in the tracking state, the image processing method provided in the embodiment of the present application also includes:

[0099] D1. For each tracking image frame captured by the camera in the same tracking state, calculate the motion offset of adjacent tracking image frames.

[0100] D2. Compare the motion offset with a preset offset condition to obtain a comparison result, and filter out image frames of interest and / or image frames of no interest from the tracking image frames according to the comparison result.

[0101] The image frames of interest here may be image frames of interest to the user, or image frames of interest to other targets. The following description will be made by taking the image frames of interest as image frames of interest to the user as an example.

[0102] Specifically, two adjacent tracking image frames among the tracking image frames captured by the camera in the tracking state are matched to obtain the matching point pairs, and then the motion offset of the two matching tracking image frames is calculated according to the offsets of all matching point pairs, such as taking the average of the offsets of all matching point pairs as the motion offset of the two matching tracking image frames. That is:

[0103]

[0104]

[0105] Wherein, (Mx, My) represents the motion offset between the a+1th tracking image frame and the ath tracking image frame, (Xki, Yki) represents the position of the i-th feature point in the a+1th tracking image frame, and (Xmj, Ymj) represents the position of the feature point that matches (Xki, Yki) in the ath tracking image frame. n represents the number of feature points that match the two tracking image frames. If the offset length is not zero, it is considered that the gimbal is rotating at this time, and the a+1th tracking image frame and the ath tracking image frame are the image frames of interest (that is, the image frames that are tracking the target are the image frames of interest, such as the image frames of interest to the user). If the offset length is zero, it is considered that the gimbal temporarily stops moving, and the tracking image frames corresponding to the temporary stop of the gimbal are image frames that the user is not interested in.

[0106] It should be noted that, before matching two adjacent tracking image frames, the two tracking image frames may be preprocessed, and the preprocessing includes: removing invalid image features and / or selecting retained image features according to feature confidence. The specific preprocessing process is the same as the preprocessing of the matching image frames and the reference image, and is not limited here.

[0107] In some embodiments, in order to further improve the speed at which the user locates the video that he wants to view, the progress bar intervals corresponding to the filtered image frames that the user is interested in and / or the image frames that the user is not interested in can be marked.

[0108] Specifically, the progress bar interval corresponding to the image frame that the user is interested in can be determined from the first type of progress bar interval. Assuming that the determined progress bar interval is the third type of progress bar interval, the identification of the third type of progress bar interval is different from the identification of the second type of progress bar interval, and is also different from the identification of the first type of progress bar interval. For example, assuming that the first type of progress bar interval on the progress bar of the video to be processed is filled with green, and the second type of progress bar interval on the progress bar of the video to be processed is filled with gray, then the third type of progress bar interval can be identified with a color different from gray and green, for example, the third type of progress bar interval can be identified with red. Figure 6 In the example, the progress bar at the top is the original progress bar corresponding to the video to be processed. The original progress bar has no markings and its corresponding playback time (or video time) is 10 minutes. Figure 6 In the figure, the progress bar at the bottom is the marked progress bar corresponding to the video to be processed. The marked progress bar uses gray (text in the figure) to mark the second type of progress bar interval, and uses green to mark the first type of progress bar interval. In addition, the third type of progress bar interval is further distinguished from the first type of progress bar interval (i.e., progress bar interval N). Figure 6 Indicated in red.

[0109] In the embodiment of the present application, since the tracking image frames are further divided into image frames corresponding to when the camera is rotating and image frames corresponding to when the camera is not rotating, and the progress bar interval of the image frames corresponding to when the camera is rotating is marked on the progress bar of the video to be processed, and the image frames corresponding to when the camera is rotating are usually image frames that are tracking the target (i.e., image frames that the user is interested in), when the user only wants to view the image frames that are tracking the target, the user can quickly locate the image frames that he wants to view from the progress bar of the video to be processed. At the same time, since the first type of progress bar interval is also marked on the progress bar of the video to be processed, when the user wants to view the image frames that track the target and view the information before and after the tracking is triggered, such as viewing the video when the target walks into the screen before the tracking is triggered and the video after the target disappears, the user can quickly determine the corresponding image frame on the progress bar of the video to be processed by marking the first type of progress bar interval.

[0110] It should be noted that the method of marking the progress bar interval corresponding to the image frame that the user is not interested in is similar to the method of marking the progress bar interval corresponding to the image frame that the user is interested in, and will not be described in detail here.

[0111] In some embodiments, considering that the global feature is a rough description of the entire image, it lacks the description of the local details of the image. Therefore, when using the global feature to match the image frames, it is easy to cause the image frames before and after the gimbal moves to have a high similarity, thereby causing the judgment result of whether the gimbal is in the reset position to be incorrect. For example, assuming that the image frame X1 is the image frame corresponding to the gimbal in the reset position, and the image frame X2 is the image frame obtained after the gimbal rotates to the right (or left) a certain angle relative to the reset position, and when the similarity of the image frame X1 and the image frame X2 is calculated based on the global feature of the image frame X1 and the global feature of the image frame X2, the obtained similarity is high, then the image frame X2 is determined to be the image frame corresponding to the gimbal in the reset position, that is, an incorrect judgment result is obtained. Considering that the local feature includes the description of the local details of the image, therefore, compared with the image similarity algorithm based on the frame difference method, the use of local features to match the image frames is not affected by the change of illumination and the slight offset of the gimbal reset. That is, in some embodiments, the image features of the embodiments of the present application include depth local features. In this case, the above step S11 includes:

[0112] The deep local features of each image frame to be matched in the above-mentioned video to be processed are obtained through a trained deep convolutional neural network, wherein the above-mentioned deep convolutional neural network includes a convolutional layer, a pooling layer, an attention mechanism layer, a feature integration layer and a fully connected layer, and the number of layers of the above-mentioned convolutional layer is greater than a preset layer number threshold.

[0113] In the embodiment of the present application, considering that a deeper network helps to extract more robust local features, the number of convolutional layers is set to be greater than a preset layer threshold. The preset layer threshold is a larger value, for example, the preset layer threshold can be set to 90. Since the number of convolutional layers of the trained deep convolutional neural network needs to be greater than 90, the number of convolutional layers can be set to 99.

[0114] In an embodiment of the present application, the pooling layer mainly processes the image features extracted by the convolution layer. When the image features include deep local features, the pooling layer includes compressing the extracted deep local features, etc.; the attention mechanism layer is mainly used to guide the attention direction of the deep convolutional neural network; the feature integration layer is used to fuse the image features output by the pooling layer; the fully connected layer is used to connect each node to all nodes in the previous layer.

[0115] In some embodiments, the number of layers corresponding to the pooling layer, the attention mechanism layer, the feature integration layer, and the fully connected layer is a smaller value. For example, the number of layers corresponding to the pooling layer, the attention mechanism layer, the feature integration layer, and the fully connected layer can be set to 1. Since the number of layers corresponding to the pooling layer, the attention mechanism layer, the feature integration layer, and the fully connected layer is a smaller value, the deep convolutional neural network of the embodiment of the present application outputs image features faster than when the number of layers is a larger value.

[0116] It should be pointed out that the deep local features of the above-mentioned reference image can also be obtained through a trained deep convolutional neural network, which will not be described in detail here.

[0117] In some embodiments, the above-mentioned deep convolutional neural network can directly train the network parameters of the convolutional layer, the fully connected layer and the attention mechanism layer included therein, or can train the network parameters according to a specified training sequence, such as by training in the following manner:

[0118] E1. Build a deep convolutional neural network to be trained.

[0119] Among them, the network structure of the deep convolutional neural network to be trained is the same as the network structure of the trained deep convolutional neural network, which will not be repeated here.

[0120] E2. Using the constructed image dataset, first train the network parameters of the convolutional layer and the fully connected layer, and then train the network parameters of the attention mechanism layer to obtain a trained deep convolutional neural network.

[0121] In the embodiment of the present application, considering that the network of the convolution layer and the fully connected layer is deep, therefore, in order to guide the optimization direction of the network and reduce invalid network training, 1) the first training first freezes the attention mechanism layer, unfreezes the convolution layer and the fully connected layer (wherein the network parameters of the pooling layer and the feature integration layer are fixed parameters and do not participate in the training), and then uses the constructed image data set for training to obtain better detection and classification effects as a basis; 2) the second training trains the attention mechanism layer separately to further guide the network's attention direction and quickly achieve training performance that meets the requirements. After the training is completed, the image frame (the image frame to be matched or the reference image) is input into the trained deep convolutional neural network to obtain its deep local features (each image frame can contain multiple deep local features, and each deep local feature can contain a deep local feature position (X, Y), a deep local feature confidence Conf, and a deep local feature vector Vector.

[0122] In some embodiments, the image dataset may be constructed by:

[0123] Based on the public landmark dataset, a tracking image dataset with multiple categories (human, car, battery car, etc.), multiple perspectives (different PTZ positions and lens magnifications), multiple scenes, multiple illuminations and resolutions based on surveillance cameras is constructed, and data augmentation operations (mirroring, rotating and cropping) are performed to obtain the image dataset of the embodiment of the present application. The image dataset obtained by the above processing can make the mining of single data features more comprehensive and closer to the real scene, which is conducive to improving the generalization ability of the obtained deep convolutional neural network.

[0124] In order to further describe the image processing method provided in the embodiment of the present application, Figure 7 Give a description.

[0125] exist Figure 7 The image processing method of the present application mainly includes three parts: data acquisition, model training and gimbal state recognition.

[0126] The data collection part includes three parts: monitoring image data collection, building a monitoring image data set based on the collected monitoring image data, and collecting landmark data based on the monitoring image data in the monitoring image data set.

[0127] Among them, in the model training part: first, the monitoring image data and the collected landmark image data are processed by augmenting the data by mirroring, rotating and cropping, so as to obtain the final image data set constructed by the embodiment of the present application, and then the final image data set is used to train the deep convolutional neural network to be trained. During the training process, the network parameters of the convolutional layer and the fully connected layer of the deep convolutional neural network are first trained, and then the attention mechanism layer of the deep convolutional neural network is trained.

[0128] Among them, in the pan-tilt state recognition part: firstly, the image frame to be matched is captured by the monitoring camera, and then the image frame to be matched is grayed, and the image frame to be matched after the graying is input into the trained deep convolutional neural network to obtain the deep local features output by the deep convolutional neural network. The deep local features with the same position as the foreground object are removed from the deep local features of the image frame to be matched (that is, invalid image features are removed), and the deep local features of the image to be matched from which the invalid image features are removed are matched with the deep local features of the sparse reference image, and the number of matched feature points is counted, and it is determined according to the statistical results whether the image frame to be matched is an image frame taken by the camera when the pan-tilt is in the reset position. If so, it indicates that the image frame to be matched is a non-tracking image frame taken by the camera in the non-tracking state, and if not, it indicates that the image frame to be matched is a tracking image frame taken by the camera in the tracking state. In addition, for two adjacent tracking image frames taken by the camera when the pan-tilt is not in the reset position, the motion offset of the two tracking image frames is calculated, and it is judged whether the camera (or pan-tilt) is rotating according to the calculated motion offset.

[0129] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0130] Embodiment 2:

[0131] Corresponding to the image processing method of the first embodiment above, Figure 8 A structural block diagram of an image processing device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0132] Reference Figure 8 The image processing device 8 includes: an image feature acquisition module 81, a target image frame determination module 82, and a tracking image frame screening module 83. Among them:

[0133] The image feature acquisition module 81 is used to acquire image features of each image frame to be matched in the video to be processed, where the video to be processed is captured by a camera.

[0134] The target image frame determination module 82 is used to determine the image frame that matches the above-mentioned reference image based on the image features of the above-mentioned image frames to be matched and the image features of the reference image, so as to obtain the target image frame, wherein the above-mentioned reference image is the image frame captured by the above-mentioned camera in a non-tracking state.

[0135] The tracking image frame filtering module 83 is used to filter out the tracking image frames taken by the camera in the tracking state from the video to be processed according to the position of the target image frame in the video to be processed, and / or to filter out the non-tracking image frames taken by the camera in the non-tracking state from the video to be processed.

[0136] In an embodiment of the present application, an image frame matching the reference image is determined by acquiring the image features of each image frame in the video to be processed and the image features of the reference image, and a target image frame is obtained. Since the reference image is an image frame captured by the camera in a non-tracking state, the image frame matching the reference image (i.e., the target image frame) is an image frame captured by the camera in a non-tracking state. Therefore, according to the position of the target image frame in the video to be processed, it is possible to filter out the tracking image frames captured by the camera in a tracking state from the video to be processed, and / or, filter out the non-tracking image frames captured by the camera in a non-tracking state from the video to be processed. Since the tracking image frames and / or the non-tracking image frames are filtered out from the video to be processed, that is, by classifying the image frames in the video to be processed, it is beneficial for the user to quickly obtain effective information based on the filtered tracking image frames and / or non-tracking image frames, thereby improving the speed at which the user obtains effective information.

[0137] In some embodiments, the image processing device 8 provided in the embodiment of the present application further includes:

[0138] Two types of progress bar interval determination modules are used to determine the progress bar interval corresponding to the tracking image frame on the progress bar of the video to be processed according to the tracking image frame to obtain the first type of progress bar interval. And\or, according to the non-tracking image frame, determine the progress bar interval corresponding to the non-tracking image frame on the progress bar of the video to be processed to obtain the second type of progress bar interval.

[0139] Two types of progress bar interval identification modules are used to identify the first type of progress bar interval and / or the second type of progress bar interval on the progress bar of the video to be processed.

[0140] In some embodiments, the image processing device 8 provided in the embodiment of the present application further includes:

[0141] The progress bar interval determination module to be corrected is used to determine the progress bar interval to be corrected, wherein the progress bar interval to be corrected includes: a first type of progress bar interval whose corresponding video duration is less than a preset duration threshold.

[0142] The progress bar interval correction module is used to correct the progress bar interval to be corrected from the first type of progress bar interval to the second type of progress bar interval.

[0143] In some embodiments, the image processing device 8 provided in the embodiment of the present application further includes:

[0144] The wireless feature elimination module is used to eliminate invalid image features of each image frame to be matched, wherein the image features located at the position of the foreground object of the above image frame to be matched are invalid image features of the above image frame to be matched.

[0145] Correspondingly, the target image frame determination module 82 is specifically used for:

[0146] An image frame matching the reference image is determined based on the image features of the image frames to be matched, from which invalid image features have been eliminated, and the image features of the reference image.

[0147] In some embodiments, the image processing device 8 provided in the embodiment of the present application further includes:

[0148] The target area determination module is used to determine the target area according to the above-mentioned feature position and a preset area radius.

[0149] The image feature retaining module is used to retain all the image features of the reference image in the target area if the number of the image features of the reference image in the target area is less than a preset number threshold.

[0150] A sorting module is used to sort the image features of the above-mentioned benchmark images in the above-mentioned target area in descending order according to the feature confidence of the above-mentioned image features if the number of image features of the above-mentioned benchmark images in the above-mentioned target area is not less than the above-mentioned number threshold, and retain the image features of the above-mentioned benchmark images with the top N order, wherein the above-mentioned N is equal to the above-mentioned number threshold.

[0151] Correspondingly, the target image frame determination module 82 is specifically used for:

[0152] An image frame matching the reference image is determined based on the image features of the image frames to be matched and the image features retained in the reference image.

[0153] In some embodiments, the image processing device 8 provided in the embodiment of the present application further includes:

[0154] The motion offset calculation module is used to calculate the motion offset of adjacent tracking image frames for each tracking image frame captured by the camera in the same tracking state.

[0155] The motion offset comparison module is used to compare the motion offset with a preset offset condition to obtain a comparison result, and filter out image frames of interest and / or image frames of no interest from the tracking image frames according to the comparison result.

[0156] In some embodiments, the above-mentioned image features include depth local features, and the above-mentioned image feature acquisition module 81 is specifically used for:

[0157] The deep local features of each image frame to be matched in the above-mentioned video to be processed are obtained through a trained deep convolutional neural network, wherein the above-mentioned deep convolutional neural network includes a convolutional layer, a pooling layer, an attention mechanism layer, a feature integration layer and a fully connected layer, and the number of layers of the above-mentioned convolutional layer is greater than a preset layer number threshold.

[0158] In some embodiments, the above deep convolutional neural network is trained in the following manner:

[0159] Build a deep convolutional neural network to be trained.

[0160] Using the constructed image dataset, the network parameters of the convolutional layer and the fully connected layer are first trained, and then the network parameters of the attention mechanism layer are trained to obtain a trained deep convolutional neural network.

[0161] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0162] Embodiment three:

[0163] Fig. 9 This is a schematic diagram of the structure of a processing device provided in one embodiment of the present application. Fig. 9 As shown, the processing device 9 of this embodiment includes: at least one processor 90 ( Fig. 9 Only one processor is shown in the figure), a memory 91, and a computer program 92 stored in the above-mentioned memory 91 and executable on at least one of the above-mentioned processors 90, wherein the above-mentioned processor 90 implements the steps in any of the above-mentioned method embodiments when executing the above-mentioned computer program 92.

[0164] The processing device 9 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The processing device may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will appreciate that Fig. 9 It is only an example of the processing device 9 and does not constitute a limitation on the processing device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0165] The processor 90 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0166] In some embodiments, the memory 91 may be an internal storage unit of the processing device 9, such as a hard disk or memory of the processing device 9. In other embodiments, the memory 91 may also be an external storage device of the processing device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the processing device 9. Further, the memory 91 may also include both an internal storage unit of the processing device 9 and an external storage device. The memory 91 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 91 may also be used to temporarily store data that has been output or is to be output.

[0167] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0168] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0169] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0170] An embodiment of the present application provides a computer program product. When the computer program product runs on a processing device, the processing device can implement the steps in the above-mentioned various method embodiments when executing the computer program product.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0172] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0174] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Obtaining image features of each image frame to be matched in a video to be processed, wherein the video to be processed is obtained by being shot by a camera; According to the image features of each image frame to be matched and the image features of the reference image, an image frame matching the reference image is determined to obtain a target image frame, wherein the reference image is an image frame captured by the camera in a non-tracking state, the visual range of the camera is adjusted by the pan-tilt head, and when the pan-tilt head is in a reset position, the camera is in a non-tracking state; when the pan-tilt head is in a non-reset position, the camera is in a tracking state; According to the position of the target image frame in the video to be processed, the tracking image frames captured by the camera in the tracking state are filtered out from the video to be processed, and / or the non-tracking image frames captured by the camera in the non-tracking state are filtered out from the video to be processed.

2. The image processing method according to claim 1, characterized in that: After filtering out the tracking image frames obtained by the camera in the tracking state from the video to be processed, or after filtering out the non-tracking image frames obtained by the camera in the non-tracking state from the video to be processed, the method further includes: According to the tracking image frame, determining the progress bar interval corresponding to the tracking image frame on the progress bar of the video to be processed to obtain a first type of progress bar interval; and\or, according to the non-tracking image frame, determining the progress bar interval corresponding to the non-tracking image frame on the progress bar of the video to be processed to obtain a second type of progress bar interval; The first type of progress bar interval and / or the second type of progress bar interval are marked on the progress bar of the video to be processed.

3. The image processing method according to claim 2, characterized in that: After obtaining the progress bar interval of the second type, the method further includes: Determine a progress bar interval to be corrected, wherein the progress bar interval to be corrected includes: a first type of progress bar interval whose corresponding video duration is less than a preset duration threshold; The progress bar interval to be corrected is corrected from the first type of progress bar interval to the second type of progress bar interval.

4. The image processing method according to claim 1, wherein: Before determining the image frame matching the reference image according to the image features of each of the image frames to be matched and the image features of the reference image, the method further includes: Eliminating invalid image features of each image frame to be matched, wherein the image features located at the position of the foreground object of the image frame to be matched are invalid image features of the image frame to be matched; The step of determining an image frame that matches the reference image according to the image features of each of the image frames to be matched and the image features of the reference image comprises: An image frame matching the reference image is determined based on the image features of each image frame to be matched, from which invalid image features are eliminated, and the image features of the reference image.

5. The image processing method according to claim 1, wherein: The image features of the reference image include feature positions and feature confidences. Before determining the image frame matching the reference image based on the image features of each image frame to be matched and the image features of the reference image, the method further includes: Determine the target area according to the characteristic position and the preset area radius; If the number of the image features of the reference image in the target area is less than a preset number threshold, retaining all the image features of the reference image in the target area; If the number of the image features of the reference image in the target area is not less than the number threshold, sorting the image features of the reference image in the target area in descending order according to the feature confidences of the image features, and retaining the top N image features of the reference image, where N is equal to the number threshold; The step of determining an image frame that matches the reference image according to the image features of each of the image frames to be matched and the image features of the reference image comprises: An image frame matching the reference image is determined based on the image features of each image frame to be matched and the image features retained in the reference image.

6. The image processing method according to claim 1, characterized in that: Also includes: For each tracking image frame captured by the camera in the same tracking state, calculating the motion offset of adjacent tracking image frames; The motion offset is compared with a preset offset condition to obtain a comparison result, and image frames of interest and / or image frames of no interest are screened out from the tracking image frames according to the comparison result.

7. The image processing method according to any one of claims 1 to 6, characterized in that: The image features include deep local features, and the step of obtaining the image features of each image frame to be matched in the video to be processed includes: The deep local features of each image frame to be matched in the video to be processed are obtained through a trained deep convolutional neural network, wherein the deep convolutional neural network includes a convolutional layer, a pooling layer, an attention mechanism layer, a feature integration layer and a fully connected layer, and the number of convolutional layers is greater than a preset layer number threshold.

8. The image processing method according to claim 7, characterized in that: The deep convolutional neural network is trained in the following way: Build a deep convolutional neural network to be trained; Using the constructed image data set, the network parameters of the convolutional layer and the fully connected layer are first trained, and then the network parameters of the attention mechanism layer are trained to obtain a trained deep convolutional neural network.

9. An image processing device, characterized in that: include: An image feature acquisition module is used to acquire image features of each image frame to be matched in a video to be processed, where the video to be processed is captured by a camera; a target image frame determination module, configured to determine an image frame that matches the reference image according to the image features of each image frame to be matched and the image features of the reference image, and obtain a target image frame, wherein the reference image is an image frame captured by the camera in a non-tracking state, the visual range of the camera is adjusted by the pan-tilt head, and when the pan-tilt head is in a reset position, the camera is in a non-tracking state; when the pan-tilt head is in a non-reset position, the camera is in a tracking state; A tracking image frame filtering module is used to filter out tracking image frames taken by the camera in a tracking state from the video to be processed according to the position of the target image frame in the video to be processed, and / or to filter out non-tracking image frames taken by the camera in a non-tracking state from the video to be processed.

10. A processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images