Synchronous frame image determination method, device, computer equipment and readable storage medium

Abstract

The embodiments of the present application provide a method, device, computer equipment and readable storage medium for determining a synchronous frame image. For the estimated synchronous frame binocular image captured by the binocular camera, the first type of feature points of the first image and the second type of feature points of the second image are first extracted, and the first type of feature points and the second type of feature points are feature point matched to obtain all basic matching pairs and target matching pairs that meet preset conditions in the basic matching pairs, wherein each basic matching pair includes a first type of feature point and a second type of feature point. Finally, a comprehensive difference value is calculated based on the target matching pair and all basic matching pairs. By comparison, if the comprehensive difference value meets the preset difference threshold, it is determined that the first image and the second image are synchronous frame images. In this way, without increasing the hardware cost, it is relatively accurate and fast to determine whether the preliminarily estimated binocular image is a synchronous frame image, thereby optimizing the synchronous detection effect of the binocular image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, computer equipment and readable storage medium for determining a synchronous frame image. Background Art

[0002] The application of binocular cameras is becoming more and more popular, and the demand for synchronous frame calibration of binocular cameras is also increasing. In order to reduce the cost of hardware, the existing binocular cameras only have a relatively cheap imaging engine (Image Signal Processing, referred to as ISP) image processing unit, which mainly puts the binocular images into the buffer at the same time after the camera exposure is completed, and reads them through a single channel, that is, reads the left image first and then the right image. When the central processing unit (Central Processing Unit, referred to as CPU) is overloaded, the reading efficiency will be affected. For example, when the left image is being read, the image data at the next moment will overwrite the unread data, and there will be a frame error phenomenon, and serious asynchronous frame situation.

[0003] It can be seen that the existing synchronization frame detection scheme has the technical problem of poor synchronization detection effect. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present application provide a method, an apparatus, a computer device and a readable storage medium for determining a synchronous frame image.

[0005] In a first aspect, an embodiment of the present application provides a method for determining a synchronous frame image, the method comprising:

[0006] Extracting first-category feature points of a first image and second-category feature points of a second image, wherein the first image and the second image are binocular images of estimated synchronous frames acquired by a binocular camera, and a pixel area of ​​the first-category feature points in the first image is the same as a pixel area of ​​the second-category feature points in the second image;

[0007] Perform feature point matching on the first-category feature points and the second-category feature points to obtain all basic matching pairs and target matching pairs that meet preset conditions among the basic matching pairs, wherein each basic matching pair includes a first-category feature point and a second-category feature point;

[0008] Calculating a comprehensive difference value based on the target matching pair and all basic matching pairs;

[0009] If the comprehensive difference value meets the preset difference threshold, it is determined that the first image and the second image are synchronized frame images.

[0010] According to a specific implementation of the present application, the step of extracting the first type of feature points of the first image and the second type of feature points of the second image includes:

[0011] determining a key pixel region from the first image and the second image;

[0012] Feature extraction is performed on the key pixel area of ​​the first image and the key pixel area of ​​the second image respectively to obtain corresponding first-category feature points and second-category feature points.

[0013] According to a specific implementation of the present application, the step of respectively extracting features from the key pixel area of ​​the first image and the key pixel area of ​​the second image to obtain corresponding first-category feature points and second-category feature points includes:

[0014] Acquire adjacent pixel points of each basic pixel point in the key pixel area of ​​the first image and the key pixel area of ​​the second image;

[0015] The first-category feature points and the second-category feature points are determined according to the pixel value of each basic pixel point and the pixel value of the corresponding adjacent pixel point.

[0016] According to a specific implementation of the present application, the step of determining the first type of feature points and the second type of feature points according to the pixel value of each basic pixel point and the pixel value of the corresponding adjacent pixel point includes:

[0017] From all adjacent pixels of each basic pixel, find the target adjacent point whose grayscale difference meets the grayscale threshold;

[0018] Determine whether the difference between the number of target adjacent points of each basic pixel and all adjacent pixel points meets the point count threshold;

[0019] Basic pixel points whose quantity difference between target adjacent points and all adjacent pixel points in the first image and the second image meets the point number threshold are respectively used as first-category feature points of the first image and second-category feature points of the second image.

[0020] According to a specific implementation of the present application, the step of performing feature point matching on the first type of feature points and the second type of feature points to obtain all basic matching pairs and target matching pairs in the basic matching pairs that meet preset conditions includes:

[0021] Calculate the descriptor of each of the first-category feature points and the descriptor of the second-category feature points;

[0022] Traversing and comparing the descriptors of each of the first-category feature points and the descriptors of each of the second-category feature points to find all basic matching pairs of descriptor matching;

[0023] Find a target matching pair in which the difference in the number of pixel rows between the first type of feature points and the second type of feature points in each basic matching pair meets the row number threshold.

[0024] According to a specific implementation of the present application, if the comprehensive difference value satisfies a preset difference threshold, the step of determining that the first image and the second image are synchronized frame images includes:

[0025] Calculate the pixel row difference between the first type of feature points and the second type of feature points in each basic matching pair;

[0026] Calculate the average value of the pixel row differences of all basic matching pairs as the pixel row difference average value;

[0027] If the comprehensive difference value is greater than or equal to the pixel row difference average value, it is determined that the first image and the second image are synchronized frame images.

[0028] According to a specific implementation of the present application, the step of calculating the comprehensive difference value based on the target matching pair and all basic matching pairs includes:

[0029] Calculating the match pair difference between the base match pair and the target match pair;

[0030] Divide the number of target matching pairs by the matching pair difference to obtain a row difference ratio;

[0031] The row difference ratio is multiplied by a ratio coefficient to obtain the comprehensive difference value.

[0032] According to a specific implementation of the present application, the value range of the line number threshold is 2 lines to 8 lines, and the value range of the proportional coefficient is 1.5-2.0.

[0033] In a second aspect, an embodiment of the present application provides a synchronous frame image determination device, the device comprising:

[0034] An extraction module, configured to extract first-category feature points of a first image and second-category feature points of a second image, wherein the first image and the second image are binocular images of estimated synchronous frames acquired by a binocular camera, and a pixel area of ​​the first-category feature points in the first image is the same as a pixel area of ​​the second-category feature points in the second image;

[0035] A matching module, used for matching the first-category feature points with the second-category feature points, and obtaining all basic matching pairs and target matching pairs that meet preset conditions among the basic matching pairs, wherein each basic matching pair includes a first-category feature point and a second-category feature point;

[0036] A calculation module, used for calculating a comprehensive difference value based on the target matching pair and all basic matching pairs;

[0037] The determination module is configured to determine that the first image and the second image are synchronous frame images if the comprehensive difference value satisfies a preset difference threshold.

[0038] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method for determining a synchronized frame image described in any one of the first aspects is executed.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed on a processor, executes the method for determining a synchronized frame image as described in any one of the first aspects.

[0040] The synchronous frame image determination method, device, computer equipment and readable storage medium provided by the present application are for the binocular image of the estimated synchronous frame collected by the binocular camera. The first type of feature points of the first image and the second type of feature points of the second image are first extracted, and the pixel area of ​​the first type of feature points in the first image is the same as the pixel area of ​​the second type of feature points in the second image. The first type of feature points and the second type of feature points are feature point matched to obtain all basic matching pairs and target matching pairs that meet the preset conditions in the basic matching pairs, wherein each basic matching pair includes a first type of feature point and a second type of feature point. Finally, the comprehensive difference value is calculated based on the target matching pair and all basic matching pairs. If the comprehensive difference value meets the preset difference threshold by comparison, it is determined that the first image and the second image are synchronous frame images. In this way, without increasing the hardware cost, it is more accurate and fast to determine whether the preliminarily estimated binocular image is a synchronous frame image, and the synchronous detection effect of the binocular image is optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present application and should not be regarded as limiting the scope of protection of the present application. In each of the drawings, similar components are numbered similarly.

[0042] Figure 1 A schematic diagram of a process of determining a synchronous frame image provided by an embodiment of the present application is shown;

[0043] Figure 2 A schematic diagram of the spatial position involved in the method for determining a synchronous frame image provided in an embodiment of the present application is shown;

[0044] Figure 3 A schematic diagram showing basic pixel points and adjacent pixel points involved in the method for determining a synchronous frame image provided in an embodiment of the present application is shown;

[0045] Figure 4 A module block diagram of a synchronous frame image determination device provided by an embodiment of the present application is shown;

[0046] Figure 5 A hardware structure diagram of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0048] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0049] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0050] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0051] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0052] Example 1

[0053] See also Figure 1, is a flow chart of a method for determining a synchronous frame image provided by an embodiment of the present application. Figure 1 As shown, the method mainly comprises the following steps:

[0054] Step S101, extracting first-category feature points of a first image and second-category feature points of a second image, wherein the first image and the second image are binocular images of estimated synchronous frames acquired by a binocular camera, and the pixel area of ​​the first-category feature points in the first image is the same as the pixel area of ​​the second-category feature points in the second image;

[0055] The synchronous frame image determination method provided in this embodiment is applied to a binocular camera integrated with the processing function of a computer device, or a computer device connected to the data of the binocular camera, and is used to determine whether the binocular image captured by the binocular camera is a synchronous frame image, mainly for the binocular image captured by the binocular camera as an estimated synchronous frame on the time axis, which is defined as the first image and the second image respectively. It should be noted that the first image and the second image here can be a group of images preliminarily estimated to be synchronous frames captured by the binocular camera at any time, usually after the computer device receives the end of the photo shooting instruction, a frame of image extracted from the first buffer area for storing the image captured by the left camera, and a frame of image extracted from the second buffer area for storing the image captured by the right camera. Usually, the image captured by the left camera in the binocular camera can be used as the first image, and the image captured by the right camera can be used as the second image. Of course, the names can also be reversed, which is only used to distinguish the reference and is not limited.

[0056] The synchronous frame image determination solution provided in this embodiment is that there will be a certain degree of overlap between the two frames of images (i.e., the first image and the second image) in the synchronous frame images captured by the binocular camera, and the pixel areas of the overlapping content on each image must satisfy certain relationship factors. Therefore, the characteristics of the pixel points of the first image and the second image can be used to distinguish whether the two frames of images are synchronous frames.

[0057] Specifically, feature points of the first image and the second image are extracted respectively, and are defined as first-category feature points and second-category feature points. To avoid the influence of pixel areas on subsequent feature point recognition and matching, the first-category feature points and the second-category feature points can be obtained from specific pixel areas in the first image and the second image, that is, the feature points are all taken from the middle area of ​​the corresponding image or from the edge area of ​​the corresponding image.

[0058] Step S102, performing feature point matching on the first-category feature points and the second-category feature points to obtain all basic matching pairs and target matching pairs that meet preset conditions among the basic matching pairs, wherein each basic matching pair includes a first-category feature point and a second-category feature point;

[0059] After obtaining the first-class feature points of the first image and the second-class feature points of the second image, each first-class feature point can be matched with each second-class feature point, that is, the first-class feature points and the second-class feature points with the same or similar pixel values ​​are searched. The first-class feature points and the second-class feature points that match the feature points are taken as a basic matching pair. The number of the first-class feature points and the second-class feature points are both multiple, and the number of basic matching pairs obtained may be one or more. It should be noted that the traversal matching method can be used in the feature point matching process, that is, each first-class feature point is matched with each second-class feature point before matching the next first-class feature point, or vice versa, each second-class feature point is matched with each first-class feature point before matching the next second-class feature point. Each first-class feature point and each second-class feature point can only belong to one basic matching pair.

[0060] After performing feature point matching to obtain basic matching pairs, it is also necessary to screen out a small number of basic matching pairs that can better characterize the characteristics of the synchronization frame from the basic matching pairs, which are defined as target matching pairs. Preset conditions can be set in advance, such as screening conditions such as a small number of row differences, a row difference within a preset range, and a descriptor matching degree of feature points higher than a certain value. Of course, other conditions that can characterize the characteristics of the synchronization frame can also be used. After obtaining all the basic matching pairs, the preset conditions can be used for screening to screen out the target matching pairs. The number of target matching pairs is less than or equal to the number of basic matching pairs, and the specific number difference can be determined by the preset conditions on which the screening is based.

[0061] Step S103, calculating a comprehensive difference value based on the target matching pair and all basic matching pairs;

[0062] After obtaining basic matching pairs that characterize similar features to a certain extent through feature point matching, and selecting target matching pairs that are closer to the features of the synchronization frame, the corresponding numerical value that can characterize the proportion or degree of proximity can be calculated based on the target matching pairs and the basic matching pairs, and the numerical value is defined as the comprehensive difference value. There are many ways to calculate the comprehensive difference value, such as directly calculating the proportion of the target matching pair in all basic matching pairs, or calculating the difference between the basic matching value and the target matching pair or the proportion of the difference, etc., without limitation.

[0063] Step S104: if the comprehensive difference value satisfies a preset difference threshold, determining that the first image and the second image are synchronized frame images.

[0064] The computer device determines a critical value that the synchronized frame image must meet based on historical experience values ​​or experimental data, which is defined as a preset difference threshold. After obtaining the comprehensive difference value corresponding to the current first image and the second image, it is determined whether the current comprehensive difference value meets the preset difference threshold of critical synchronization. If it does, the first image and the second image are determined to be synchronized frame images. Of course, considering that the actual critical value obtained may be the threshold that critically meets the synchronized frame, it may also be the threshold that critically does not meet the synchronized frame. If the critical value obtained is the threshold that does not meet the synchronized frame, other values ​​except the critical value can be used as the preset critical threshold, or the non-satisfaction of the critical value can be used as the condition for determining it as a synchronized frame, etc. If the result obtained proves that the current binocular image is a synchronized frame image, it can be retained. If it is an unsynchronized frame image, the current image can be deleted.

[0065] In addition, based on this embodiment, according to a specific implementation of the present application, before the step of extracting the first type of feature points of the first image and the second type of feature points of the second image, the method may further include:

[0066] The binocular camera is adjusted so that the baseline and the epipolar line are parallel to each other.

[0067] This embodiment takes into account the influence of the coordinate system calibration of the binocular camera itself on the subsequent feature point matching. Before feature point extraction and matching, the binocular camera is first calibrated and corrected so that the baseline and epipolar line of the binocular camera are parallel to each other.

[0068] The specific adjustment process can be as follows Figure 2 As shown in the figure, the binocular camera is first calibrated to obtain the intrinsic and extrinsic parameters of the binocular camera. The new image coordinate system is recalculated using the calibrated intrinsic and extrinsic parameters so that the 5 axes of the two images are on the same horizontal line and the epipolar line is parallel to the baseline.

[0069] Baseline correction process: Align the binocular cameras to the same observation plane so that the pixel rows on the binocular cameras are strictly aligned. R and T can be obtained through camera calibration, p_l and p_r are the two pixel points corresponding to the P space point on the binocular camera, and after baseline correction, they will appear on the same horizontal line, that is, on the same pixel row of the image.

[0070] In order to transform the pole of the binocular camera to infinity so that the pole line is horizontally aligned, a rotation matrix is ​​created. First, the 5-axis rotation is performed to rotate the 5-axis to the same direction as the baseline. The new 5-axis after rotation is O_l-O_r. The rotation vector

[0071] To determine the new Y axis after rotation, it only needs to be orthogonal to e1, that is:

[0072] e2=e1×(Rl [0, 0, 1] T +t l ).

[0073] The new Z axis is perpendicular to e1 and e2 and satisfies the right-hand rule, that is:

[0074] e3=e1×e2.

[0075] The final rotation matrix: R new =[e1, e2, e3], and then get the rotation matrix of each mesh:

[0076]

[0077]

[0078] Where R_l and R_r are the rotation matrices from world coordinates to camera coordinates, and the final result is the rotation matrix of the binocular image from the old image coordinate system to the new image coordinate system. After the baseline correction is completed, ensure that the points p_l and p_r of the same aerial point P on the binocular image are on the same horizontal line.

[0079] Of course, a conventional geometric calibration method for a binocular camera may also be used to adjust the baseline of the binocular camera to be parallel to the epipolar line, without limitation.

[0080] The method for determining a synchronous frame image provided by the present application is to first extract the first type of feature points of the first image and the second type of feature points of the second image for the binocular image of the estimated synchronous frame collected by the binocular camera, wherein the pixel area of ​​the first type of feature points in the first image is the same as the pixel area of ​​the second type of feature points in the second image. The first type of feature points and the second type of feature points are feature point matched to obtain all basic matching pairs and target matching pairs that meet preset conditions in the basic matching pairs, wherein each basic matching pair includes a first type of feature point and a second type of feature point. Finally, a comprehensive difference value is calculated based on the target matching pair and all basic matching pairs, and if the comprehensive difference value meets the preset difference threshold by comparison, it is determined that the first image and the second image are synchronous frame images. In this way, without increasing the hardware cost, it is more accurate and faster to determine whether the preliminarily estimated binocular image is a synchronous frame image, thereby optimizing the synchronous detection effect of the binocular image.

[0081] On the basis of the above embodiments, several specific implementation methods will be combined below to further define the steps of feature point extraction, feature point matching, etc.

[0082] According to a specific implementation of the present application, the step of extracting the first type of feature points of the first image and the second type of feature points of the second image in S102 may specifically include:

[0083] determining a key pixel region from the first image and the second image;

[0084] Feature extraction is performed on the key pixel area of ​​the first image and the key pixel area of ​​the second image respectively to obtain corresponding first-category feature points and second-category feature points.

[0085] In this embodiment, when extracting feature points, key pixel areas are first determined in the first image and the second image respectively, and the pixel areas of the key pixel areas on the two images are the same, that is, both are middle areas or both are edge areas. When extracting feature points later, feature points can be extracted according to pixel values ​​from all basic pixel points contained in the key pixel areas. This can effectively avoid the influence of different selection areas of feature points on the accuracy of synchronous frame judgment.

[0086] Exemplarily, considering that the main photographed object is generally positioned in the central area of ​​the field of view during shooting, in order to ensure the reference of the extracted feature points as much as possible, the middle area of ​​the first image and the second image can be used as the key pixel area, for example, the area where the middle 50% of the pixel rows are located is used as the key pixel area. Of course, the area where the most central 3-10 pixel rows are located can also be selected as the key pixel area, without limitation. When extracting feature points, only 20 feature points can be extracted in the middle pixel area, such as ORB (Oriented Fast and Rotated Brief) feature points. ORB is used to quickly create feature vectors for key points in the image, and these feature vectors can be used to identify objects in the image. Not extracting feature points across the entire image and only extracting 20 points can effectively ensure that the calculation results are simple and fast, and further ensure the real-time nature of synchronous frame recognition.

[0087] Furthermore, the step of respectively extracting features from the key pixel area of ​​the first image and the key pixel area of ​​the second image to obtain corresponding first-category feature points and second-category feature points includes:

[0088] Acquire adjacent pixel points of each basic pixel point in the key pixel area of ​​the first image and the key pixel area of ​​the second image;

[0089] The first-category feature points and the second-category feature points are determined according to the pixel value of each basic pixel point and the pixel value of the corresponding adjacent pixel point.

[0090] In this embodiment, when extracting the first type of feature points and the second type of feature points, considering that there are usually differences in pixel values ​​between feature points and adjacent pixels, when extracting feature points, the pixel values ​​of each basic pixel point and its adjacent pixels can be referred to to determine whether the basic pixel point is the pixel point to be extracted. In the specific implementation, in order to avoid unnecessary calculation steps, each pixel point in the key pixel area of ​​the first image and the second image is defined as a basic pixel point. When distinguishing which basic pixel points may be the first type of feature points and the second type of feature points, the critical pixel points of these basic pixel points can be extracted and defined as the adjacent pixel points of the basic pixel point.

[0091] Specifically, the step of determining the first type of feature points and the second type of feature points according to the pixel value of each basic pixel point and the pixel value of the corresponding adjacent pixel point may include:

[0092] From all adjacent pixels of each basic pixel, find the target adjacent point whose grayscale difference meets the grayscale threshold;

[0093] Determine whether the difference between the number of target adjacent points of each basic pixel and all adjacent pixel points meets the point count threshold;

[0094] Basic pixel points whose quantity difference between target adjacent points and all adjacent pixel points in the first image and the second image meets the point number threshold are respectively used as first-category feature points of the first image and second-category feature points of the second image.

[0095] When extracting feature points, the grayscale value of each basic pixel is used to determine whether it is a feature point. In this embodiment, first find out some specific adjacent points whose grayscale difference meets the grayscale threshold among all adjacent pixels of each basic pixel, and define them as target adjacent points. Then, further determine whether the difference between the target adjacent points and all adjacent pixels meets the point count threshold, and then use the basic pixel points that meet the point count threshold as feature points. For example, Figure 3 As shown in the figure, as long as there are 12 consecutive points in the 16 points in the neighborhood of point P that are different from point P, then point P is considered to be a feature point. The feature point extraction method based on the gray value of the reference pixel can effectively improve the calculation speed.

[0096] According to a specific implementation of the present application, the step of performing feature point matching on the first type of feature points and the second type of feature points to obtain all basic matching pairs and target matching pairs in the basic matching pairs that meet preset conditions may include:

[0097] Calculate the descriptor of each of the first-category feature points and the descriptor of the second-category feature points;

[0098] Traversing and comparing the descriptors of each of the first-category feature points and the descriptors of each of the second-category feature points to find all basic matching pairs of descriptor matching;

[0099] Find a target matching pair in which the difference in the number of pixel rows between the first type of feature points and the second type of feature points in each basic matching pair meets the row number threshold.

[0100] After extracting the first type of feature points of the first image and the second type of feature points of the second image, the descriptor of each feature point can be calculated and the feature points can be matched. The first type of feature points and the second type of feature points whose descriptors satisfy the basic matching form a basic matching pair. After obtaining all the basic matching pairs, the preset conditions can be used to filter out the target matching pairs. Here, the basic matching pairs whose line number difference is less than or equal to the line number threshold are used as the target matching pairs to avoid possible mismatching.

[0101] The final judgment step, that is, the step of determining that the first image and the second image are synchronized frame images if the comprehensive difference value meets the preset difference threshold, includes:

[0102] Calculate the pixel row difference between the first type of feature points and the second type of feature points in each basic matching pair;

[0103] Calculate the average value of the pixel row differences of all basic matching pairs as the pixel row difference average value;

[0104] If the comprehensive difference value is greater than or equal to the pixel row difference average value, it is determined that the first image and the second image are synchronized frame images.

[0105] In specific implementation, the pixel row difference between the two feature points of each basic matching pair is calculated, and after accumulating and averaging, the pixel row difference average value 5_means is calculated. The comprehensive difference value calculated in the above step is then compared with the pixel row difference average value. If the comprehensive difference value is greater than or equal to the pixel row difference average value, it is determined that the first image and the second image are synchronized frame images.

[0106] Furthermore, the step of calculating the comprehensive difference value based on the target matching pair and all basic matching pairs may include:

[0107] Calculating the match pair difference between the base match pair and the target match pair;

[0108] Divide the number of target matching pairs by the matching pair difference to obtain a row difference ratio;

[0109] The row difference ratio is multiplied by a ratio coefficient to obtain the comprehensive difference value.

[0110] This implementation provides a more specific rule for calculating the comprehensive difference value, and determines whether the current frame is an asynchronous frame according to the difference in row coordinates of all matching points. During implementation, the row difference threshold is 3 rows.

[0111] The number of basic matching pairs is marked as good_matches, and the number of target matching pairs whose pixel row difference is less than the row number threshold is marked as min_matchs. Then the comprehensive formula for determining whether it is a synchronized frame image is as follows:

[0112]

[0113] If the current binocular image satisfies the above formula, it can be determined that the binocular image is a synchronized frame image. The mean takes into account the global basic matching pairs. The larger the number of target matching pairs min_matchs, the higher the credibility of the current frame, and the larger the threshold, and vice versa.

[0114] In specific implementation, the value range of the line number threshold is 2 lines to 8 lines, and the value range of the proportional coefficient is 1.5 to 2.0. According to several experiments, the line number threshold can be 3 lines, and the proportional coefficient can be 1.75 to obtain a higher detection accuracy.

[0115] In summary, the synchronous frame image determination method provided in the embodiment of the present application only takes 1-3ms in the process of detecting the synchronous frame, which can effectively ensure the real-time performance of binocular camera image acquisition. At the same time, the detection noise can be effectively filtered out with flexible judgment conditions, so as to accurately eliminate the asynchronous frames.

[0116] Example 2

[0117] See also Figure 4 , is a module block diagram of a synchronous frame image determination device provided in an embodiment of the present application. Figure 4 As shown, the synchronous frame image determination device 400 mainly includes:

[0118] An extraction module 401 is used to extract first-category feature points of a first image and second-category feature points of a second image, wherein the first image and the second image are binocular images of estimated synchronous frames acquired by a binocular camera, and a pixel area of ​​the first-category feature points in the first image is the same as a pixel area of ​​the second-category feature points in the second image;

[0119] A matching module 402 is used to perform feature point matching on the first-category feature points and the second-category feature points to obtain all basic matching pairs and target matching pairs that meet preset conditions among the basic matching pairs, wherein each basic matching pair includes a first-category feature point and a second-category feature point;

[0120] A calculation module 403, configured to calculate a comprehensive difference value based on the target matching pair and all basic matching pairs;

[0121] The determination module 404 is configured to determine that the first image and the second image are synchronous frame images if the comprehensive difference value satisfies a preset difference threshold.

[0122] According to a specific implementation of the present application, the extraction module 401 is used to:

[0123] determining a key pixel region from the first image and the second image;

[0124] Feature extraction is performed on the key pixel area of ​​the first image and the key pixel area of ​​the second image respectively to obtain corresponding first-category feature points and second-category feature points.

[0125] According to a specific implementation of the present application, the extraction module 401 is used to:

[0126] Acquire adjacent pixel points of each basic pixel point in the key pixel area of ​​the first image and the key pixel area of ​​the second image;

[0127] The first-category feature points and the second-category feature points are determined according to the pixel value of each basic pixel point and the pixel value of the corresponding adjacent pixel point.

[0128] According to a specific implementation of the present application, the extraction module 401 is specifically used for:

[0129] From all adjacent pixels of each basic pixel, find the target adjacent point whose grayscale difference meets the grayscale threshold;

[0130] Determine whether the difference between the number of target adjacent points of each basic pixel and all adjacent pixel points meets the point count threshold;

[0131] Basic pixel points whose quantity difference between target adjacent points and all adjacent pixel points in the first image and the second image meets the point number threshold are respectively used as first-category feature points of the first image and second-category feature points of the second image.

[0132] According to a specific implementation of the present application, the matching module 402 is used to:

[0133] Calculate the descriptor of each of the first-category feature points and the descriptor of the second-category feature points;

[0134] Traversing and comparing the descriptors of each of the first-category feature points and the descriptors of each of the second-category feature points to find all basic matching pairs of descriptor matching;

[0135] Find a target matching pair in which the difference in the number of pixel rows between the first type of feature points and the second type of feature points in each basic matching pair meets the row number threshold.

[0136] According to a specific implementation of the present application, the determination module 404 is used to:

[0137] Calculate the pixel row difference between the first type of feature points and the second type of feature points in each basic matching pair;

[0138] Calculate the average value of the pixel row differences of all basic matching pairs as the pixel row difference average value;

[0139] If the comprehensive difference value is greater than or equal to the pixel row difference average value, it is determined that the first image and the second image are synchronized frame images.

[0140] According to a specific implementation of the present application, the calculation module 403 is used to:

[0141] Calculating the match pair difference between the base match pair and the target match pair;

[0142] Divide the number of target matching pairs by the matching pair difference to obtain a row difference ratio;

[0143] The row difference ratio is multiplied by a ratio coefficient to obtain the comprehensive difference value.

[0144] According to a specific implementation of the present application, the value range of the line number threshold is 2 lines to 8 lines, and the value range of the proportional coefficient is 1.5-2.0.

[0145] In addition, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method for determining a synchronous frame image provided in the above embodiment is executed.

[0146] In addition, an embodiment of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program runs the method for determining a synchronous frame image provided by the above embodiment on a processor.

[0147] The synchronous frame image determination device, computer device and readable storage medium provided by the present application, for the binocular image of the estimated synchronous frame collected by the binocular camera, first extract the first type of feature points of the first image and the second type of feature points of the second image, the pixel area of ​​the first type of feature points in the first image is the same as the pixel area of ​​the second type of feature points in the second image. The first type of feature points and the second type of feature points are feature point matched to obtain all basic matching pairs and target matching pairs that meet the preset conditions in the basic matching pairs, wherein each basic matching pair includes a first type of feature point and a second type of feature point. Finally, the comprehensive difference value is calculated based on the target matching pair and all basic matching pairs, and if the comprehensive difference value meets the preset difference threshold by comparison, the first image and the second image are determined to be synchronous frame images. In this way, without increasing the hardware cost, it is more accurate and fast to determine whether the preliminarily estimated binocular image is a synchronous frame image, and the synchronous detection effect of the binocular image is optimized. The specific implementation process of the synchronous frame image determination device, computer device and readable storage medium provided can refer to the specific implementation process of the above-mentioned method embodiment, which will not be repeated here.

[0148] Specifically, Figure 5 As shown, it is a schematic diagram of the structure of a computer device for implementing various embodiments of the present application. Figure 5 As shown, the computer device 500 includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510, and a power supply 511. Those skilled in the art will understand that Figure 5 The computer device structure shown in the figure does not constitute a limitation on the computer device, and the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present application, the computer device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0149] It should be understood that in the embodiment of the present application, the radio frequency unit 501 can be used for receiving and sending signals during information transmission or calls. Specifically, after receiving downlink data from the base station, it is sent to the processor 510 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 501 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 501 can also communicate with the network and other devices through a wireless communication system.

[0150] The computer device provides users with wireless broadband Internet access through the network module 502, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0151] The audio output unit 503 can convert the audio data received by the RF unit 501 or the network module 502 or stored in the memory 509 into an audio signal and output it as sound. Moreover, the audio output unit 503 can also provide audio output related to a specific function performed by the computer device 500 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 503 includes a speaker, a buzzer, a receiver, etc.

[0152] The input unit 504 is used to receive audio or video signals. The input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The graphics processor 5041 processes the image data of a static picture or video obtained by an image capture computer device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be played as a video on the display unit 506. The image frame processed by the graphics processor 5041 can be stored in the memory 509 (or other storage medium) or sent via the radio frequency unit 501 or the network module 502. The microphone 5042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 501 in the case of a telephone call mode.

[0153] The computer device 500 also includes at least one sensor 505, which at least includes the barometer mentioned in the above embodiment. In addition, the sensor 505 can also be other sensors such as light sensors, motion sensors and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 5061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 5061 and / or the backlight when the computer device 500 moves to the ear. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the computer device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, knocking), etc.; the sensor 505 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0154] The display unit 506 is used for video playback of information input by the user or information provided to the user. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0155] The user input unit 507 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the computer device. Specifically, the user input unit 507 includes a touch panel 5071 and other input devices 5072. The touch panel 5071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 5071 or near the touch panel 5071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 5071 may include two parts: a touch detection computer device and a touch controller. Among them, the touch detection computer device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection computer device, converts it into the coordinates of the contact point, and then sends it to the processor 510, receives the command sent by the processor 510 and executes it. In addition, the touch panel 5071 can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 5071, the user input unit 507 may also include other input devices 5072. Specifically, other input devices 5072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail here.

[0156] Furthermore, the touch panel 5071 may be overlaid on the display panel 5061. When the touch panel 5071 detects a touch operation on or near it, it transmits the information to the processor 510 to determine the type of the touch event. Then, the processor 510 provides a corresponding visual output on the display panel 5061 according to the type of the touch event. Figure 5 In the figure, the touch panel 5071 and the display panel 5061 are used as two independent components to realize the input and output functions of the computer device. However, in some embodiments, the touch panel 5071 and the display panel 5061 can be integrated to realize the input and output functions of the computer device, which is not limited here.

[0157] The interface unit 508 is an interface for connecting an external computer device to the computer device 500. For example, the external computer device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a computer device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 508 may be used to receive input (e.g., data information, power, etc.) from an external computer device and transmit the received input to one or more elements within the computer device 500 or may be used to transmit data between the computer device 500 and an external computer device.

[0158] The memory 509 can be used to store software programs and various data. The memory 509 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 509 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0159] The processor 510 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 509 and calling data stored in the memory 509, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. The processor 510 may include one or more processing units; preferably, the processor 510 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 510.

[0160] The computer device 500 may also include a power supply 511 (such as a battery) for supplying power to various components. Preferably, the power supply 511 may be logically connected to the processor 510 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0161] In addition, the computer device 500 includes some functional modules not shown, which will not be described in detail here.

[0162] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0164] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0165] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for determining a synchronous frame image, characterized in that: The method comprises: Extracting first-category feature points of a first image and second-category feature points of a second image, wherein the first image and the second image are binocular images of estimated synchronous frames acquired by a binocular camera, and a pixel area of ​​the first-category feature points in the first image is the same as a pixel area of ​​the second-category feature points in the second image; Perform feature point matching on the first-category feature points and the second-category feature points to obtain all basic matching pairs and target matching pairs that meet preset conditions among the basic matching pairs, wherein each basic matching pair includes a first-category feature point and a second-category feature point; Calculating a comprehensive difference value based on the target matching pair and all basic matching pairs; If the comprehensive difference value meets the preset difference threshold, it is determined that the first image and the second image are synchronized frame images.

2. The method according to claim 1, characterized in that The step of extracting the first type of feature points of the first image and the second type of feature points of the second image comprises: determining a key pixel region from the first image and the second image; Feature extraction is performed on the key pixel area of ​​the first image and the key pixel area of ​​the second image respectively to obtain corresponding first-category feature points and second-category feature points.

3. The method according to claim 2, characterized in that The step of respectively extracting features from the key pixel area of ​​the first image and the key pixel area of ​​the second image to obtain corresponding first-category feature points and second-category feature points comprises: Acquire adjacent pixel points of each basic pixel point in the key pixel area of ​​the first image and the key pixel area of ​​the second image; The first-category feature points and the second-category feature points are determined according to the pixel value of each basic pixel point and the pixel value of the corresponding adjacent pixel point.

4. The method according to claim 3, characterized in that The step of determining the first type of feature points and the second type of feature points according to the pixel value of each basic pixel point and the pixel value of the corresponding adjacent pixel point comprises: From all adjacent pixels of each basic pixel, find the target adjacent point whose grayscale difference meets the grayscale threshold; Determine whether the difference between the number of target adjacent points of each basic pixel and all adjacent pixel points meets the point count threshold; Basic pixel points whose quantity difference between target adjacent points and all adjacent pixel points in the first image and the second image meets the point number threshold are respectively used as first-category feature points of the first image and second-category feature points of the second image.

5. The method according to any one of claims 1 to 4, characterized in that The step of matching the first type of feature points with the second type of feature points to obtain all basic matching pairs and target matching pairs that meet preset conditions in the basic matching pairs includes: Calculate the descriptor of each of the first-category feature points and the descriptor of the second-category feature points; Traversing and comparing the descriptors of each of the first-category feature points and the descriptors of each of the second-category feature points to find all basic matching pairs of descriptor matching; Find a target matching pair in which the difference in the number of pixel rows between the first type of feature points and the second type of feature points in each basic matching pair meets the row number threshold.

6. The method according to claim 5, characterized in that The step of determining that the first image and the second image are synchronized frame images if the comprehensive difference value meets a preset difference threshold comprises: Calculate the pixel row difference between the first type of feature points and the second type of feature points in each basic matching pair, wherein the pixel row difference is the difference in the number of rows between the pixel row where the first type of feature points and the pixel row where the second type of feature points are located in each basic matching pair; Calculate the average value of the pixel row differences of all basic matching pairs as the pixel row difference average value; If the comprehensive difference value is greater than or equal to the pixel row difference average value, it is determined that the first image and the second image are synchronized frame images.

7. The method according to claim 5, characterized in that The step of calculating the comprehensive difference value based on the target matching pair and all basic matching pairs includes: Calculating the match pair difference between the base match pair and the target match pair; Divide the number of target matching pairs by the matching pair difference to obtain a row difference ratio; The row difference ratio is multiplied by a ratio coefficient to obtain the comprehensive difference value.

8. The method according to claim 7, characterized in that The value range of the line number threshold is 2 lines to 8 lines, and the value range of the proportional coefficient is 1.5-2.

0.

9. A synchronous frame image determination device, characterized in that: The device comprises: An extraction module, configured to extract first-category feature points of a first image and second-category feature points of a second image, wherein the first image and the second image are binocular images of estimated synchronous frames acquired by a binocular camera, and a pixel area of ​​the first-category feature points in the first image is the same as a pixel area of ​​the second-category feature points in the second image; A matching module, used for matching the first-category feature points with the second-category feature points, and obtaining all basic matching pairs and target matching pairs that meet preset conditions among the basic matching pairs, wherein each basic matching pair includes a first-category feature point and a second-category feature point; A calculation module, used for calculating a comprehensive difference value based on the target matching pair and all basic matching pairs; The determination module is configured to determine that the first image and the second image are synchronous frame images if the comprehensive difference value satisfies a preset difference threshold.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is run by the processor, the method for determining a synchronized frame image according to any one of claims 1 to 8 is executed.

11. A computer-readable storage medium, characterized in that: The device stores a computer program, which executes the method for determining a synchronized frame image according to any one of claims 1 to 8 when the computer program is run on a processor.

Images