A video processing method, apparatus and device

Abstract

Embodiments of the present specification disclose a video processing method, device and equipment, the method comprises: acquiring a first image and a second image in a to-be-processed video collected by a roadside perception device; determining a target region outside a region where an object with a moving tendency is located in the first image and the second image; determining a plurality of first feature points of the target region in the first image and a plurality of second feature points of the target region in the second image; obtaining offset information between the first image and the second image based on position information of the plurality of first feature points and position information of the plurality of second feature points; aligning the second image with the first image based on the offset information to obtain a processed video. In this embodiment, the second image is aligned with the first image based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, and then the processed video is obtained, which eliminates the nonlinear deformation of the image in the video and improves the collection effect of the to-be-processed video.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video processing method, apparatus and device. Background Technology

[0002] In the context of cloud-controlled connected vehicles and autonomous driving, a large number of roadside units (RSU) and roadside computing units (RCU) have emerged.

[0003] When roadside units and roadside computing units are in operation, they collect environmental information about the road, such as video data of vehicles and pedestrians. This video data can be uploaded to the cloud to generate instructions to assist vehicles in achieving autonomous driving. However, since roadside units and roadside computing units are generally deployed on poles on both sides of the road, factors such as wind causing the poles to sway, or thermal disturbances affecting the roadside units and computing units, can cause nonlinear deformation of the images in the video, thus affecting the video acquisition quality. Summary of the Invention

[0004] This specification provides a video processing method, apparatus, and device to improve video acquisition performance.

[0005] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0006] This specification provides an embodiment of a video processing method, including:

[0007] Acquire a first image and a second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images from different frames in the video to be processed;

[0008] Identify objects with a tendency to move in the first image and the second image, and determine target areas outside the areas where the objects with a tendency to move in the first image and the second image are located;

[0009] Multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image are determined; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image;

[0010] Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, offset information between the first image and the second image is obtained;

[0011] Based on the offset information, the second image is aligned with the first image to obtain the processed video.

[0012] Optionally, identifying objects with a tendency to move in the first image and the second image, and determining a target region outside the region where the objects with a tendency to move in the first image and the second image are located, specifically includes:

[0013] Using an object recognition model, a first object with a tendency to move in the first image and a second object with a tendency to move in the second image are identified; the object recognition model is trained based on training samples, which include images labeled with objects with a tendency to move.

[0014] Determine the first position information of the first object in the first image and the second position information of the second object in the second image;

[0015] Based on the first location information, a first target region outside the region where the first object is located in the first image is determined; based on the second location information, a second target region outside the region where the second object is located in the second image is determined.

[0016] Optionally, determining multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image specifically includes:

[0017] Determine elements of a first preset type in the target region of the first image, and use at least some pixels and / or at least some sub-pixels of the elements of the first preset type as the first feature points; the first preset type includes at least one of road surface, guardrail and building.

[0018] Determine elements of a second preset type in the target region of the second image, and use at least some pixels and / or at least some sub-pixel points in the elements of the second preset type as the second feature points; the second preset type is the same as the first preset type.

[0019] Optionally, after determining multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image, the method further includes:

[0020] Based on the color information and / or texture information of the plurality of first feature points and the plurality of second feature points, the plurality of first feature points and the plurality of second feature points are matched to obtain a plurality of feature point matching pairs;

[0021] Calculate the distance between the first feature point and the second feature point in each of the feature point matching pairs;

[0022] Remove the first and second feature points corresponding to the feature point matching pairs whose distance is greater than or equal to the threshold from the plurality of first feature points and the plurality of second feature points.

[0023] Optionally, determining multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image specifically includes:

[0024] Using the SURF algorithm, SIFT algorithm, or ORB algorithm, extract multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image;

[0025] or:

[0026] At least two of the SURF, SIFT or ORB algorithms are used to extract feature points of the target region in the first image to obtain at least two sets of first feature points; the feature points in each set of first feature points are used as the first feature points.

[0027] At least two of the following algorithms are used to extract feature points of the target region in the second image: SURF, SIFT, or ORB, to obtain at least two sets of second feature points; the feature points in each set of second feature points are used as the second feature points.

[0028] Optionally, aligning the second image with the first image based on the offset information to obtain the processed video specifically includes:

[0029] Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, a nonlinear transformation matrix of the plurality of second feature points relative to the plurality of first feature points is obtained;

[0030] Based on the nonlinear transformation matrix, the second image is aligned with the first image to obtain the processed video.

[0031] Optionally, the first image is the first frame of the video to be processed; the second image is the remaining frames of the video to be processed, excluding the first frame.

[0032] Optionally, the object with a tendency to move includes at least one of vehicles and pedestrians.

[0033] This specification provides an embodiment of a video processing apparatus, comprising:

[0034] The acquisition module is used to acquire a first image and a second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images of different frames in the video to be processed.

[0035] The recognition module is used to identify objects with a tendency to move in the first image and the second image, and to determine the target area outside the area where the objects with a tendency to move in the first image and the second image are located;

[0036] The determination module is used to determine multiple first feature points of a target region in the first image and multiple second feature points of a target region in the second image; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image;

[0037] The offset information acquisition module is used to obtain offset information between the first image and the second image based on the position information of the plurality of first feature points and the position information of the plurality of second feature points;

[0038] An alignment module is used to align the second image with the first image based on the offset information to obtain the processed video.

[0039] This specification provides an embodiment of a video processing device, comprising:

[0040] At least one processor; and,

[0041] A memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0043] Acquire a first image and a second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images from different frames in the video to be processed;

[0044] Identify objects with a tendency to move in the first image and the second image, and determine target areas outside the areas where the objects with a tendency to move in the first image and the second image are located;

[0045] Multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image are determined; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image;

[0046] Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, offset information between the first image and the second image is obtained;

[0047] Based on the offset information, the second image is aligned with the first image to obtain the processed video.

[0048] One embodiment of this specification achieves the following beneficial effects: This embodiment aligns the second image with the first image using the position information of multiple first feature points and multiple second feature points to obtain the processed video, thereby eliminating nonlinear deformation of the image in the video and improving the acquisition effect of the video to be processed. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating a video processing method provided in an embodiment of this specification;

[0051] Figure 2 This is a schematic diagram of the area where a first object is located, provided in an embodiment of this specification.

[0052] Figure 3 This is a schematic diagram of the area where a second object is located, provided in an embodiment of this specification.

[0053] Figure 4 This is a schematic diagram of an image before alignment, provided as an embodiment of this specification.

[0054] Figure 5 This is a schematic diagram of an image after alignment, provided as an embodiment of this specification.

[0055] Figure 6 This is a schematic diagram illustrating the changes in an image before and after alignment, provided in an embodiment of this specification.

[0056] Figure 7 A schematic diagram of a video processing apparatus provided in the embodiments of this specification;

[0057] Figure 8 This is a schematic diagram of a video processing device provided in an embodiment of this specification. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of one or more embodiments of this specification clearer, the technical solutions of one or more embodiments of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of one or more embodiments of this specification.

[0059] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0060] In existing technologies, since roadside units and roadside computing units are generally deployed on poles on both sides of the road, the poles may sway due to factors such as wind, or the roadside units and roadside computing units may be subject to thermal disturbances, which may cause nonlinear deformation of each frame of the video, that is, different areas of the image may undergo different deformations.

[0061] The nonlinear deformation caused by the shaking of the acquisition device manifests as different deformations in areas of the scene at different distances from the acquisition device. Similarly, thermal disturbance causes nonlinear deformation, where different types of elements in the image are heated differently, leading to different deformations in areas representing those elements. These nonlinear deformations in video images negatively impact the video acquisition quality.

[0062] To address the shortcomings of existing technologies, this solution provides the following embodiments:

[0063] Figure 1 This is a flowchart illustrating a video processing method provided in an embodiment of this specification. From a programming perspective, the entity executing the process can be a program or application client hosted on an application server or in the cloud. Figure 1 As shown, the method may include the following steps:

[0064] Step 102: Acquire the first image and the second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images of different frames in the video to be processed.

[0065] The roadside sensing device may include a roadside unit and / or a roadside computing unit.

[0066] In this embodiment, the first image can be the first frame of the video to be processed, and the second image can be any of the remaining frames in the video to be processed, excluding the first frame. Alternatively, the video to be processed can be divided into multiple segments, where the first image can be the first frame of each segment, and the second image can be any of the remaining frames in each segment.

[0067] Step 104: Identify objects with a tendency to move in the first image and the second image, and determine the target area outside the area where the objects with a tendency to move in the first image and the second image are located.

[0068] Among them, objects with a tendency to move can be vehicles and / or pedestrians.

[0069] Specifically, object recognition models or algorithms can be used to identify objects with a tendency to move in the first and second images, without making any specific limitations here.

[0070] Step 106: Determine multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image.

[0071] Images possess scale invariance, meaning that image features remain unchanged under scale transformations and are unaffected by scale. In this embodiment, scale transformations can include nonlinear deformations. That is, feature points in this embodiment can be points in the image that remain unchanged when nonlinear deformations occur.

[0072] Furthermore, because objects with a tendency to move exhibit this tendency, the position of such objects within their respective frames in the video to be processed may vary significantly. In other words, the positions of feature points on objects with a tendency to move can differ considerably between two different frames, thus affecting the alignment of the two images. Therefore, this embodiment extracts feature points from the target region outside the area where the object with a tendency to move is located in the images of different frames.

[0073] Step 108: Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, obtain the offset information between the first image and the second image.

[0074] Since the feature points in this embodiment are points with constant scale in the target region outside the area where the moving object is located in the image, the position information of multiple first feature points and multiple second feature points can reflect the offset information between the first image and the second image.

[0075] Furthermore, the offset information between the first image and the second image can be obtained based on the difference between the position information of multiple first feature points and the position information of multiple second feature points.

[0076] Step 110: Based on the offset information, align the second image with the first image to obtain the processed video.

[0077] Aligning the second image with the first image can be done by aligning the second image to the first image or by aligning the first image to the second image; there is no limitation on this.

[0078] For example, by aligning the second image with the first image, we can obtain the aligned second image and thus the processed video.

[0079] This embodiment obtains the offset information between the first image and the second image based on the position information of multiple first feature points and multiple second feature points. Then, based on the offset information, the second image is aligned with the first image to obtain the processed video, thus eliminating the influence of the shaking or thermal disturbance of the roadside sensing device on the video acquisition effect.

[0080] It should be understood that the order of some steps in the methods described in one or more embodiments of this specification may be interchanged according to actual needs, or some steps may be omitted or deleted.

[0081] based on Figure 1 In addition to the method described herein, this specification also provides some specific implementation schemes of the method, which will be described below.

[0082] In this embodiment, an object recognition model can be used to identify a first object with a tendency to move in a first image and a second object with a tendency to move in a second image. The object recognition model is trained based on training samples, which include images labeled with objects exhibiting a tendency to move.

[0083] After identifying the first object and the second object, the first position information of the first object in the first image and the second position information of the second object in the second image are then determined.

[0084] Based on the first location information, a first target region outside the region where the first object is located in the first image is determined; based on the second location information, a second target region outside the region where the second object is located in the second image is determined.

[0085] For example, the minimum bounding rectangle of the first object can be determined based on the first position information. This minimum bounding rectangle can be used as the area where the first object is located, and the area where the second object is located can be determined in the same way.

[0086] Figure 2 This is a schematic diagram of the area where a first object is located, provided in an embodiment of this specification. Figure 2 As shown, the regions where the first object is located are regions A, B, and C (for illustrative purposes, not all regions are labeled in the figure). The regions in the first image other than regions A, B, and C are the first target regions.

[0087] Figure 3 This is a schematic diagram of the area where a second object is located, provided in an embodiment of this specification. Figure 3 As shown, the regions where the second object is located are regions D, E, F, and G (not all regions are labeled in the figure for illustrative purposes). The regions in the second image other than regions D, E, F, and G are the second target regions.

[0088] After determining the target region in the first image and the target region in the second image, it is necessary to determine multiple feature points in the target region.

[0089] This embodiment can utilize the SURF (SpeededUp RobustFeatures) algorithm, the SIFT (Scale-invariant feature transform) algorithm, or the ORB (Oriented FAST and Rotated BRIEF) algorithm to extract multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image.

[0090] In one embodiment, at least two of the SURF, SIFT, or ORB algorithms can be used to extract feature points of the target region in the first image to obtain at least two sets of first feature points; and the feature points in each set of first feature points are used as first feature points.

[0091] Similarly, at least two of the SURF, SIFT, or ORB algorithms can be used to extract feature points of the target region in the second image to obtain at least two sets of second feature points; the feature points in each set of second feature points are then used as second feature points.

[0092] Using at least two of the SURF, SIFT, or ORB algorithms to extract feature points can yield more feature points.

[0093] The positions of feature points on objects with a tendency to move can change significantly in two images from different frames, affecting the alignment of the two images. Therefore, this embodiment extracts feature points from the target region outside the area where the object with a tendency to move is located in the images from different frames. Similarly, within the target region, there are elements whose feature point positions are easily changed and those whose positions are not easily changed. For example, the positions of feature points on elements such as leaves and long traffic poles are easily changed, while the positions of feature points on elements such as roads, guardrails, and buildings are not easily changed. Therefore, this embodiment can extract feature points from elements such as roads, guardrails, and buildings. Specifically: determining multiple first feature points in the target region of the first image and multiple second feature points in the target region of the second image specifically includes:

[0094] Determine elements of a first preset type in the target region of the first image, and use at least some pixels and / or at least some sub-pixels of the elements of the first preset type as the first feature points; the first preset type includes at least one of road surface, guardrail and building.

[0095] Determine elements of a second preset type in the target region of the second image, and use at least some pixels and / or at least some sub-pixel points in the elements of the second preset type as the second feature points; the second preset type is the same as the first preset type.

[0096] In this embodiment, after extracting the first feature point and the second feature point, the extracted first feature point and the second feature point can be filtered to obtain more accurate feature points. That is, after determining multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image, the method further includes:

[0097] Based on the color information and / or texture information of the plurality of first feature points and the plurality of second feature points, the plurality of first feature points and the plurality of second feature points are matched to obtain a plurality of feature point matching pairs;

[0098] Calculate the distance between the first feature point and the second feature point in each of the feature point matching pairs;

[0099] Remove the first and second feature points corresponding to the feature point matching pairs whose distance is greater than or equal to the threshold from the plurality of first feature points and the plurality of second feature points.

[0100] In practical applications, shaking of the acquisition device can cause different deformations in areas of the scene at different distances from the acquisition device in the image. Additionally, different types of elements in the image are heated differently; for example, black objects on a road are heated more than non-black objects, or if the road is black asphalt, the black asphalt will absorb heat, resulting in a higher surface temperature than the area above the road. These different heating characteristics of different types of elements also cause different deformations in areas of different types of elements in the image. To eliminate this non-linear deformation, in this embodiment, aligning the second image with the first image based on the offset information to obtain the processed video specifically includes:

[0101] Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, a nonlinear transformation matrix of the plurality of second feature points relative to the plurality of first feature points is obtained;

[0102] Based on the nonlinear transformation matrix, the second image is aligned with the first image to obtain the processed video.

[0103] In this embodiment, the process of obtaining the nonlinear transformation matrix may include:

[0104] Based on the positional information of multiple first feature points and multiple second feature points, a sparse transformation matrix R is generated:

[0105]

[0106] Where m is the number of pixels and / or subpixels in the height direction of the image, and n is the number of pixels and / or subpixels in the width direction of the image; for r 11 -r mn For any pixel or subpixel in the matrix, if it does not belong to a feature point, the value is to be filled; otherwise, the value is represented by the offset information, which is determined by the offset of the feature point and the feature point it matches in the x-axis and y-axis directions.

[0107] Then, based on the values ​​represented by the offset information in the sparse transformation matrix R, the values ​​to be filled in the sparse transformation matrix R are filled using first-order or second-order interpolation to obtain the nonlinear transformation matrix T:

[0108]

[0109] A nonlinear transformation matrix T can represent the transformation of an image. Aligning the second image with the first image can be understood as using the nonlinear transformation matrix T as the transformation to transform the second image and then align it with the first image.

[0110] Figure 4This is a schematic diagram of an image before alignment, provided as an embodiment of this specification. Figure 5 This is a schematic diagram illustrating an image alignment process provided in an embodiment of this specification. Figure 6 This is a schematic diagram illustrating the changes in an image before and after alignment, provided in an embodiment of this specification. Figure 6 Through Figure 4 and Figure 5 Obtained by subtracting pixels one by one. Visual comparison. Figure 4 and Figure 5 Although there is no difference, to more clearly illustrate the differences between the two figures, we will use... Figure 6 It is expressed in the form of. Through Figure 6 It can be seen that the method of this embodiment changes the image before and after alignment.

[0111] In this embodiment, by aligning the second image with the first image based on the nonlinear transformation matrix of multiple second feature points in the second image relative to multiple first feature points in the first image, deformation occurring in different regions of the image can be eliminated, thereby improving the acquisition effect of the video to be processed.

[0112] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods. Figure 7 The embodiments provided in this specification correspond to Figure 1 A schematic diagram of a video processing device. (See diagram below.) Figure 7 As shown, the device may include:

[0113] The acquisition module 702 is used to acquire a first image and a second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images of different frames in the video to be processed;

[0114] The recognition module 704 is used to recognize objects with a tendency to move in the first image and the second image, and to determine a target area outside the area where the objects with a tendency to move in the first image and the second image are located;

[0115] The determining module 706 is used to determine multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image;

[0116] The offset information acquisition module 708 is used to obtain offset information between the first image and the second image based on the position information of the plurality of first feature points and the position information of the plurality of second feature points;

[0117] Alignment module 710 is used to align the second image with the first image based on the offset information to obtain the processed video.

[0118] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0119] Figure 8 The embodiments provided in this specification correspond to Figure 2 A schematic diagram of the structure of a video processing device. (See diagram below.) Figure 8 As shown, device 800 may include:

[0120] At least one processor 810; and,

[0121] A memory 830 is communicatively connected to the at least one processor 810; wherein,

[0122] The memory 830 stores instructions 820 that can be executed by the at least one processor 810, the instructions 820 being executed by the at least one processor 810 to enable the at least one processor 810 to:

[0123] Acquire a first image and a second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images from different frames in the video to be processed;

[0124] Identify objects with a tendency to move in the first image and the second image, and determine target areas outside the areas where the objects with a tendency to move in the first image and the second image are located;

[0125] Multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image are determined; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image;

[0126] Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, offset information between the first image and the second image is obtained;

[0127] Based on the offset information, the second image is aligned with the first image to obtain the processed video.

[0128] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, for... Figure 8As the video processing device shown is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0129] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0130] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0131] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0132] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0133] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0139] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0141] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0143] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A video processing method, characterized in that, include: Acquire a first image and a second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images from different frames in the video to be processed; Identify objects with a tendency to move in the first image and the second image, and determine target areas outside the areas where the objects with a tendency to move in the first image and the second image are located; Multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image are determined; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image; Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, offset information between the first image and the second image is obtained; Based on the offset information, the second image is aligned with the first image to obtain the processed video; The step of aligning the second image with the first image based on the offset information to obtain the processed video specifically includes: Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, a nonlinear transformation matrix of the plurality of second feature points relative to the plurality of first feature points is obtained; Based on the nonlinear transformation matrix, the second image is aligned with the first image to obtain the processed video.

2. The method according to claim 1, characterized in that, The step of identifying objects with a tendency to move in the first image and the second image, and determining target areas outside the areas where the objects with a tendency to move are located in the first image and the second image, specifically includes: Using an object recognition model, a first object with a tendency to move in the first image and a second object with a tendency to move in the second image are identified; the object recognition model is trained based on training samples, which include images labeled with objects with a tendency to move. Determine the first position information of the first object in the first image and the second position information of the second object in the second image; Based on the first location information, a first target region outside the region where the first object is located in the first image is determined; based on the second location information, a second target region outside the region where the second object is located in the second image is determined.

3. The method according to claim 1, characterized in that, The determination of multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image specifically includes: Determine elements of a first preset type in the target region of the first image, and use at least some pixels and / or at least some sub-pixels of the elements of the first preset type as the first feature points; the first preset type includes at least one of road surface, guardrail and building. Determine elements of a second preset type in the target region of the second image, and use at least some pixels and / or at least some sub-pixel points in the elements of the second preset type as the second feature points; the second preset type is the same as the first preset type.

4. The method according to claim 1 or 3, characterized in that, After determining multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image, the method further includes: Based on the color information and / or texture information of the plurality of first feature points and the plurality of second feature points, the plurality of first feature points and the plurality of second feature points are matched to obtain a plurality of feature point matching pairs; Calculate the distance between the first feature point and the second feature point in each of the feature point matching pairs; Remove the first and second feature points corresponding to the feature point matching pairs whose distance is greater than or equal to the threshold from the plurality of first feature points and the plurality of second feature points.

5. The method according to claim 1, characterized in that, The determination of multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image specifically includes: Using the SURF algorithm, SIFT algorithm, or ORB algorithm, extract multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image; or: At least two of the SURF, SIFT or ORB algorithms are used to extract feature points of the target region in the first image to obtain at least two sets of first feature points; the feature points in each set of first feature points are used as the first feature points. At least two of the following algorithms are used to extract feature points of the target region in the second image: SURF, SIFT, or ORB, to obtain at least two sets of second feature points; the feature points in each set of second feature points are used as the second feature points.

6. The method according to claim 1, characterized in that, The first image is the first frame of the video to be processed; the second image is the remaining frames of the video to be processed, excluding the first frame.

7. The method according to claim 1, characterized in that, The objects with a tendency to move include at least one of vehicles and pedestrians.

8. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire a first image and a second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images of different frames in the video to be processed. The recognition module is used to identify objects with a tendency to move in the first image and the second image, and to determine the target area outside the area where the objects with a tendency to move in the first image and the second image are located; The determination module is used to determine multiple first feature points of a target region in the first image and multiple second feature points of a target region in the second image; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image; The offset information acquisition module is used to obtain offset information between the first image and the second image based on the position information of the plurality of first feature points and the position information of the plurality of second feature points; An alignment module is used to align the second image with the first image based on the offset information to obtain a processed video. Specifically, aligning the second image with the first image based on the offset information to obtain the processed video includes: Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, a nonlinear transformation matrix of the plurality of second feature points relative to the plurality of first feature points is obtained; Based on the nonlinear transformation matrix, the second image is aligned with the first image to obtain the processed video.

9. A video processing device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Acquire a first image and a second image from the video to be processed collected by the roadside sensing device; the first image and the second image are images from different frames in the video to be processed; Identify objects with a tendency to move in the first image and the second image, and determine target areas outside the areas where the objects with a tendency to move in the first image and the second image are located; Multiple first feature points of the target region in the first image and multiple second feature points of the target region in the second image are determined; the multiple first feature points are points with invariant scale in the first image, and the multiple second feature points are points with invariant scale in the second image; Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, offset information between the first image and the second image is obtained; Based on the offset information, the second image is aligned with the first image to obtain the processed video. Specifically, aligning the second image with the first image based on the offset information to obtain the processed video includes: Based on the position information of the plurality of first feature points and the position information of the plurality of second feature points, a nonlinear transformation matrix of the plurality of second feature points relative to the plurality of first feature points is obtained; Based on the nonlinear transformation matrix, the second image is aligned with the first image to obtain the processed video.

Images