A method, medium and computing device for detecting changes in the shooting pose of a camera

Abstract

Multiple embodiments disclosed in this specification provide a method, a medium, and a computing device for detecting changes in the shooting pose of a camera. An object that remains stationary in the real scene captured by the camera is used as a reference object. The first original image obtained by the camera capturing the real scene in a preset pose and the second original image obtained by the camera capturing the real scene in the current pose are respectively acquired. The first original image is processed using target segmentation technology to obtain a first contour image highlighting the contour of the reference object. The second original image is processed based on target segmentation technology to obtain a corresponding second contour image highlighting the contour of the reference object. If the degree of offset of the reference object contour in the second contour image relative to the reference object contour in the first contour image exceeds a preset degree, it can be determined that the shooting pose of the camera has changed, that is, the current pose is different from the preset pose.

Description

Technical Field

[0001] Multiple embodiments of this specification relate to the field of information technology, and in particular, to a method, medium, and computing device for detecting changes in the shooting pose of a camera. Background Art

[0002] The shooting pose of a camera refers to the posture, focal length, etc. adopted by the camera during shooting. In some scenarios, the camera has a preset shooting pose (i.e., the preset pose of the camera), and the transaction party can calculate real-scene related data based on the image obtained by the camera shooting the real scene in the preset pose. For example, the transaction party can calculate the position coordinates of the vehicles driving on the road based on the traffic road image shot by the camera under the preset pose parameters.

[0003] However, some situations may cause changes in the shooting pose of the camera. For example, situations such as deformation of the camera's installation components or vibration of the road section where the camera is installed may cause the shooting pose of the camera to change passively. Also, for example, the background control system of the camera actively changes the shooting pose of the camera in response to a user instruction.

[0004] If the shooting pose of the camera changes without the knowledge of the transaction party, it will directly affect the accuracy of the real-scene related data calculated by the transaction party. Therefore, the transaction party urgently needs a technical solution that can detect whether the shooting pose of the camera has changed. Summary of the Invention

[0005] Multiple embodiments of this specification provide a method, medium, and computing device for detecting changes in the shooting pose of a camera, so that the transaction party can detect whether the shooting pose of the camera has changed.

[0006] According to a first aspect of multiple embodiments of this specification, a method for detecting changes in the shooting pose of a camera is proposed, including:

[0007] Obtain a first original image obtained by the camera shooting a real scene in a preset pose, and obtain a second original image obtained by the camera shooting the real scene in the current pose; wherein, the real scene includes a reference object that remains stationary;

[0008] Based on the first original image, obtain a first contour image highlighting the contour of the reference object; based on the second original image, obtain a corresponding second contour image highlighting the contour of the reference object;

[0009] Analyze the degree of offset of the reference object contour in the second contour image relative to the reference object contour in the first contour image as the degree of offset corresponding to the second contour image;

[0010] If the deviation degree corresponding to the second contour image exceeds a first preset degree, it is determined that the current pose is different from the preset pose.

[0011] According to the second aspect of multiple embodiments of this specification, a method for correcting a coordinate transformation matrix is proposed. Among them, the first coordinate transformation matrix is used to convert the pixel point coordinates in the image obtained by the camera shooting the traffic scene in the preset pose into longitude and latitude coordinates; the road surface in the traffic scene includes multiple lane lines arranged continuously at a fixed interval. The method includes:

[0012] After determining that the current pose of the camera is different from the preset pose, based on the first original image, a third contour image highlighting multiple continuous lane line contours on the road surface is obtained; based on the second original image, a fourth contour image highlighting multiple continuous lane line contours on the road surface is obtained;

[0013] Collect M third pixel points from the boundaries of multiple continuous lane line contours in the third contour image, and collect M fourth pixel points from the boundaries of multiple continuous lane line contours in the fourth contour image; among them, the M third pixel points and the M fourth pixel points correspond one by one, and the relative position of any first pixel point on the boundary of the lane line contour in the first contour image is the same as the relative position of its corresponding second pixel point on the boundary of the lane line contour in the second contour image; M>1;

[0014] Determine the second coordinate transformation matrix according to the pixel coordinate transformation relationship between each group of corresponding third pixel points and fourth pixel points;

[0015] Based on the second coordinate transformation matrix, correct the first coordinate transformation matrix so that the corrected first coordinate transformation matrix is used to convert the pixel point coordinates in the image obtained by the camera shooting the traffic scene in the current pose into longitude and latitude coordinates.

[0016] According to the third aspect of multiple embodiments of this specification, a computing device is proposed, including a memory and a processor; the memory is used to store computer instructions that can run on the processor, and the processor is used to implement the method described in the first aspect or the second aspect when executing the computer instructions.

[0017] According to the fourth aspect of multiple embodiments of this specification, a computer-readable storage medium is proposed, on which a computer program is stored, and the program implements the method described in the first aspect or the second aspect when executed by the processor.

[0018] In the above technical solution, an object that remains stationary in the real scene captured by the camera is used as a reference object. The first original image obtained by the camera capturing the real scene in a preset pose and the second original image obtained by the camera capturing the real scene in the current pose are respectively acquired. The first original image is processed using target segmentation technology to obtain a first contour image highlighting the contour of the reference object, and the second original image is processed based on target segmentation technology to obtain a corresponding second contour image highlighting the contour of the reference object.

[0019] The first contour image and the second contour image share the same image coordinate system. Therefore, the greater the degree of offset of the reference object contour in the second contour image relative to the reference object contour in the first contour image, the higher the possibility that the shooting pose of the camera has changed. If the degree of offset of the reference object contour in the second contour image relative to the reference object contour in the first contour image exceeds the preset degree, it can be determined that the shooting pose of the camera has changed, that is, the current pose is different from the preset pose.

[0020] Through the above technical solution, for the party that calculates real-scene related data based on the images captured by the camera, it can detect whether the shooting pose of the camera is likely to change only relying on the images captured by the camera. Brief Description of the Drawings

[0021] Figure 1 An exemplary process of a method for detecting a change in the shooting pose of a camera is provided.

[0022] Figure 2 An exemplary schematic diagram of the process for detecting a change in the shooting pose of a camera is provided.

[0023] Figure 3 An exemplary process of a method for correcting a coordinate transformation matrix is provided.

[0024] Figure 4 It is a schematic structural diagram of a computer-readable storage medium provided by the present disclosure.

[0025] Figure 5 It is a schematic structural diagram of a computing device provided by the present disclosure.

[0026] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. The number of any element in the drawings is for illustration rather than limitation, and any naming is only for distinction without any limiting meaning. Detailed Description of the Embodiments

[0027] To enable those skilled in the art to better understand the technical solutions in this specification, the following will clearly and completely describe the technical solutions in the embodiments of this specification in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this specification without making creative efforts shall fall within the scope of protection of this specification.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method do not necessarily need to be executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0029] In practical applications, the transaction party needs to calculate the real-scene related data based on the images obtained by the camera shooting the real scene. If the shooting pose of the camera changes (that is, the current shooting pose of the camera is no longer the preset pose), then the images obtained by the camera shooting the real scene in the current pose are very likely to be mismatched with the image calculation method configured by the transaction party based on the preset pose, resulting in inaccurate real-scene related data calculated by the transaction party. Therefore, it is necessary for the transaction party to detect whether the shooting pose of the camera has changed.

[0030] However, the transaction party is usually not the controller or manager of the camera, does not understand the control protocol of the camera, and it is not easy to obtain the control signal of the camera. Therefore, it is very difficult to know whether the shooting pose of the camera has actually changed.

[0031] Based on this, it is necessary to provide a technical solution for the transaction party to detect the change of the shooting pose of the camera without relying on the control protocol of the camera.

[0032] Specifically, an object that remains stationary in the real scene captured by the camera can be used as a reference object. The first original image obtained by the camera shooting the real scene in the preset pose and at least one second original image obtained by the camera shooting the real scene in the current pose are respectively acquired. The first original image is processed using the target segmentation technology to obtain a first contour image highlighting the contour of the reference object. Each second original image is processed based on the target segmentation technology to obtain a corresponding second contour image highlighting the contour of the reference object.

[0033] The first contour image and the second contour image share the same image coordinate system. Therefore, the greater the offset degree of the reference object contour in the second contour image relative to the reference object contour in the first contour image, the higher the possibility that the shooting pose of the camera has changed. If the offset degree of the reference object contour in the second contour image relative to the reference object contour in the first contour image exceeds the preset degree, it can be determined that the shooting pose of the camera has changed, that is, the current pose is different from the preset pose.

[0034] Through the above technical solution, for the party that calculates the real-scene related data based on the images captured by the camera, it can detect whether the shooting pose of the camera may change only relying on the images captured by the camera.

[0035] The above technical solution will be described in detail below with reference to the accompanying drawings.

[0036] Figure 1 An exemplary process of a method for detecting the change of the shooting pose of a camera is provided, including the following steps:

[0037] S100: Obtain the first original image captured by the camera for the real scene in the preset pose, and obtain the second original image captured by the camera for the real scene in the current pose.

[0038] The camera in the present disclosure may refer to a camera deployed in a real scene (i.e., the real scene) for shooting the real scene. For example, the camera can monitor the real scene and shoot a monitoring video of the real scene, and the monitoring video consists of several frames of images captured by the camera for the real scene.

[0039] Specifically, the camera can be a PTZ camera. PTZ is the abbreviation of Pan / Tilt / Zoom. Pan represents the movement of the camera in the horizontal direction, that is, the rotation of the camera; Tilt represents the movement of the camera in the vertical direction, that is, the up and down pitching of the camera lens; Zoom represents zooming, that is, adjusting the focal length of the camera. It is easy to understand that PTZ can define the shooting pose of the camera.

[0040] The above real scene can be, for example, a traffic scene, an office building scene, etc.

[0041] Generally, the camera has a preset shooting pose (i.e., the preset pose), and the party configures its adopted image calculation method based on the preset pose of the camera so as to calculate accurate real-scene related parameters according to the images captured by the camera in the preset pose.

[0042] For example, the transaction party can calculate the position coordinates of the vehicles traveling on the road based on the traffic road image captured by the camera at the preset pose parameters, so as to track the driving route of the vehicles. For another example, the transaction party can calculate the driving direction of the vehicles traveling on the road based on the traffic road image captured by the camera at the preset pose parameters, so as to determine whether the vehicle is driving in the wrong direction.

[0043] If the shooting pose of the camera changes (no longer the preset pose), then the transaction party needs to know that this change has occurred in order to adjust its image calculation method in a timely manner, so that the adjusted image calculation method is adapted to the changed shooting pose.

[0044] The first original image described in this disclosure may refer to a frame of image obtained by the camera shooting the real scene at the preset pose. The second original image described in this disclosure may refer to a frame of image obtained by the camera shooting the real scene at the current pose.

[0045] The reason for taking the objects that remain stationary in the real scene as the reference objects is that the spatial relationship between such reference objects and the shooting pose of the camera is relatively fixed. By using the property that the first original image and the second original image share the image coordinate system, the offset degree between the reference object contours in the first original image and the second original image can be compared, which can indirectly reflect the difference degree between the preset pose and the current pose of the camera.

[0046] In some embodiments, the above real scene may be a traffic scene, for example, and the above reference object may be a road surface. It is easy to understand that the camera is usually arranged on one side or both sides of the road surface. For example, the camera can be arranged on the street lamp poles on both sides of the road surface. Some road surfaces have double lanes with opposite driving directions. In this case, the reference object in the first original image and the reference object in the second original image should usually be the same road surface in a certain driving direction.

[0047] In some embodiments, only one second original image can be obtained, and it is analyzed and judged whether the shooting pose of the camera changes based on the first original image and the second original image. In other embodiments, at least two (for example, 3) second original images can be obtained, and the first original image and each second original image are taken as a group, and it is comprehensively analyzed and judged whether the shooting pose of the camera changes according to each group of images. Correspondingly, if the offset degree corresponding to each second contour image exceeds the first preset degree, it can be determined that the current pose is different from the preset pose. The comprehensive judgment result obtained by analyzing based on multiple second original images can better exclude the situation that "the camera pose actually does not change, but special circumstances occur in the shooting environment, resulting in misjudgment in the analysis of the second original image taken".

[0048] S102: Based on the first original image, obtain a first contour image highlighting the contour of the reference object; based on each second original image, obtain a corresponding second contour image highlighting the contour of the reference object.

[0049] Step S102 is usually implemented based on object segmentation technology. The object segmentation technology can be, for example, semantic segmentation or instance segmentation. The object segmentation technology is used to segment the contour area of a specific object from an image. The contour area of the object is also called the mask (MASK) of the object, which is essentially the highlighted area in a binary image. It should be noted here that in the field of object segmentation technology of images, MASK is the contour area of the object segmented from the image. Since the contour area of the object usually does not contain pixel point information, like covering a mask for the object, the contour area of the object is also called MASK. It is easy to understand that when the reference object is the object to be segmented, a contour image highlighting the contour of the reference object can be obtained.

[0050] S104: Analyze the offset degree of the reference object contour in the second contour image relative to the reference object contour in the first contour image as the offset degree corresponding to this second contour image.

[0051] S106: If the offset degree corresponding to the second contour image exceeds the first preset degree, determine that the current pose is different from the preset pose.

[0052] In some embodiments, if the offset degree corresponding to the second contour image is lower than or equal to the first preset degree, it can be determined that the current pose has not changed or the change is not obvious relative to the preset pose.

[0053] Based on the property that the first contour image and the second contour image share the image coordinate system, those skilled in the art can easily think of various ways to analyze the offset degree of the reference object contour in the second contour image relative to the reference object contour in the first contour image. The embodiments provided in this disclosure are only examples.

[0054] Perform an Intersection Of Union (IOU) calculation for the reference object contour in the second contour image and the reference object contour in the first contour image; in the case where the IOU calculation result is greater than the first ratio, determine that the offset degree corresponding to this second contour image is lower than or equal to the preset degree.

[0055] Assume that the first area is A and the second area is B. Then, the IOU calculation formula for the intersection of the first area and the second area is:

[0056] IOU(A,B)=(A∩B) / (A∪B - A∩B).

[0057] In some embodiments, if the IOU calculation result is less than or equal to a first ratio (e.g., 90%), it can be determined that the deviation degree corresponding to the second contour image exceeds a first preset degree.

[0058] In other embodiments, if the IOU calculation result is less than or equal to the first ratio, for the accuracy of the judgment, no conclusion can be drawn temporarily. Instead, the overlapping degree between the boundary of the reference object contour in the second contour image and the boundary of the reference object contour in the first contour image is further analyzed. If the overlapping degree is lower than a second preset degree, it is determined that the deviation degree corresponding to the second contour image exceeds the first preset degree.

[0059] In addition, if the overlapping degree is higher than or equal to the second preset degree, the relationship between the overlapping degree and a third preset degree can be further judged, and the third preset degree is greater than the second preset degree. If the overlapping degree is higher than the third preset degree, it can be determined that the deviation degree corresponding to the second contour image is less than or equal to the first preset degree (i.e., the deviation degree is not high enough).

[0060] If the overlapping degree is higher than or equal to the second preset degree and lower than or equal to the third preset degree, it means that the overlapping degree is neither very high nor very low, and it is not easy to determine whether this overlapping degree can indicate that the deviation degree corresponding to the second contour image is high enough.

[0061] Therefore, the following auxiliary means can be adopted for further determination:

[0062] An image usually includes not only the reference objects that are stationary in the real scene, but also a set of background objects (one or more background objects) that remain stationary in the real scene. For example, an image obtained by a camera shooting a traffic scene includes not only the road surface (reference object), but also background objects such as green belts and street lamp poles. The background objects that remain stationary in the real scene also do not change their positions with different shooting times, and the pixel point distribution structure contained in the image including the background objects is correlated with the shooting pose of the camera. If the pixel point distribution structures contained in the background object set images in the first original image and the second original image are relatively similar, it can be explained that the shooting poses of the camera when shooting the first original image and the second original image are relatively close.

[0063] It should be noted that the real scene may also include moving objects, such as vehicles driving on a traffic road. The moving objects in the real scene will change their positions over different shooting times. Therefore, the pixel point distribution structure contained in the moving object images has no correlation with the shooting pose of the camera. Therefore, when analyzing the similarity of the pixel point distribution structures contained in the background object images in the first original image and the second original image, it is actually based on the first background image highlighting the background object set image and the second background image highlighting the background object set image to calculate the similarity of the pixel point distribution structures of the first background image and the second background image.

[0064] Specifically, if the overlapping degree is higher than or equal to the second preset degree and lower than or equal to the third preset degree, then a first background image highlighting the background object set image obtained based on the first original image and a second background image highlighting the corresponding background object set image obtained based on the second original image corresponding to the second contour image can be obtained.

[0065] Calculate the similarity of the pixel point distribution structures of the first background image and the second background image as the similarity corresponding to the second contour image. If the similarity corresponding to the second background image exceeds the specified similarity, it is determined that the offset degree corresponding to the second contour image is lower than or equal to the first preset degree. If the similarity corresponding to the second background image is lower than or equal to the specified similarity, it is determined that the offset degree corresponding to the second contour image exceeds the first preset degree.

[0066] Assume that the first part of the image is denoted as x and the second part of the image is denoted as y. Then, an exemplary formula for calculating the Structure Similarity Index Measure (SSIM) value of the pixel point distribution of x and y is provided here:

[0067]

[0068] Among them, u represents the average value of the pixel values of the corresponding image, c 1 、c 2 are constants, σ xy represents the covariance of the pixel value matrices of x and y, σ x represents the standard deviation of the pixel value matrix of x, and σ y represents the standard deviation of the pixel value matrix of y.

[0069] If the similarity between each second background image and the first background image is lower than or equal to the specified similarity, it can be determined that the current pose is different from the preset pose. If the similarity between the second part of the image extracted from any one of the second original images and the first part of the image exceeds the preset similarity, it can be determined that the current pose is different from the preset pose.

[0070] It should be noted that in the technical solution provided by the present disclosure, all background objects in the real scene are fused into an overall feature to characterize the pixel distribution structure. Therefore, there is no need to rely on manual annotation of different types of background objects in the image (to facilitate the selection of the most prominent background object), and there is no need to consider the situation where environmental factors such as lighting, weather, and occlusion may cause the background objects in the image to be unclear, and the implementation of the solution is more robust. In addition, all background objects in the real scene are fused into an overall feature to characterize the pixel distribution structure. Although it is not refined enough, the overall feature has only a general degree of correlation with the camera's shooting posture (not a very strong correlation). However, the above method is only an auxiliary means based on the main means such as reference object contour IOU calculation and reference object contour boundary overlap comparison. It is a further analysis based on the main means that have ensured a high accuracy. Therefore, the detection result of whether the camera posture has changed is still relatively accurate.

[0071] In addition, as to how to analyze the degree of overlap between the boundary of the reference object contour in the second contour image and the boundary of the reference object contour in the first contour image, those skilled in the art can easily think of multiple ways based on the property that the first contour image and the second contour image share an image coordinate system. The present disclosure exemplarily provides one way as follows:

[0072] N second pixel points can be sampled from the boundary of the reference object contour in the second contour image, and N first pixel points can be sampled from the boundary of the reference object contour in the first contour image; wherein the N second pixel points correspond one-to-one to the N first pixel points, the first pixel points and the second pixel points having a corresponding relationship form a pixel point combination, and the relative position of the first pixel point on the boundary of the reference object contour in the first contour image in the same pixel point combination is the same as the relative position of the corresponding second pixel point on the boundary of the reference object contour in the second contour image; N>1.

[0073] Then, the relative distance between the first pixel point and the second pixel point in each pixel point combination in the image coordinate system can be calculated, and the proportion of pixel point combinations whose relative distance is less than the preset distance in all pixel point combinations can be determined as the degree of overlap between the boundary of the reference object contour in the second contour image and the boundary of the reference object contour in the first contour image.

[0074] After determining that the current posture of the camera is different from the preset posture, the current posture can be inferred according to the first original image, the second original image and the preset posture, so as to calculate the real scene related data based on the current posture. For example, the transaction party can adjust the image calculation method based on the current posture so that the adjusted image calculation method is adapted to the current posture.

[0075] If it is determined that the current pose of the camera is the same as the preset pose, it indicates that the shooting pose of the camera has not changed, and there is no need to adjust the image calculation method.

[0076] In some embodiments, after determining that the current pose is different from the preset pose, at least one third original image captured by the camera of the real scene in the current pose can be obtained. The third original image here may refer to the image obtained by re-shooting the real scene in the current pose after determining that the shooting pose of the camera has changed.

[0077] Based on the third original image, a corresponding third contour image highlighting the contour of the reference object can be obtained. Then, the offset degree of the reference object contour in the third contour image relative to the reference object contour in the first contour image can be analyzed as the offset degree corresponding to the third contour image. Then, if the offset degree corresponding to the third contour image exceeds the first preset degree, it can be determined that the current pose has been restored to the preset pose.

[0078] That is to say, according to the technical idea adopted in the technical solution of the present disclosure, it can be further detected whether the shooting pose of the camera has been restored to the preset pose after the shooting pose changes. Of course, in embodiments where the number of second original images is multiple, correspondingly, when it is necessary to further detect whether the shooting pose of the camera has been restored to the preset pose after the shooting pose changes, it is also necessary to make a judgment based on multiple third original images. If the offset degree corresponding to each third contour image exceeds the first preset degree, it can be determined that the current pose has been restored to the preset pose.

[0079] Figure 2 An exemplary process diagram for detecting the change of the shooting pose of the camera is provided. As Figure 2 shown, taking the example of only obtaining one second original image, it is easy to understand that in embodiments where multiple second original images are obtained, the process shown in Figure 2 is executed for each second original image. After obtaining the result of the temporary pose change based on each second original image, the actual pose change can be determined. When the result that the pose has not changed is obtained based on any second original image, it can be determined that the actual pose has not changed.

[0080] In addition, in a traffic scenario, sometimes it is necessary to convert the pixel coordinates in the image captured by the camera into longitude and latitude coordinates. For example, in a traffic scenario, a camera is deployed, and the camera is used to monitor multiple driving position points of a vehicle. The pixel point coordinates of the multiple driving position points in the image are converted into longitude and latitude coordinates, and the longitude and latitude coordinates of these driving position points form a driving route.

[0081] Converting the pixel coordinates in the image into longitude and latitude coordinates depends on a coordinate conversion matrix (referred to as the first coordinate conversion matrix in this article, which belongs to the homography matrix). The first coordinate conversion matrix is determined according to the shooting pose of the camera. Usually, the first coordinate conversion matrix is default determined according to the preset pose of the camera. If the current pose of the camera is no longer the preset pose, then the pixel coordinates in the image captured by the camera in the current pose no longer match the first coordinate conversion matrix, which easily leads to inaccurate converted longitude and latitude coordinates.

[0082] Therefore, it is necessary to correct the first coordinate conversion matrix so that the corrected first coordinate conversion matrix can more accurately convert the pixel coordinates in the image captured by the camera in the current pose of the traffic scene into longitude and latitude coordinates.

[0083] A solution for correcting the first coordinate conversion matrix provided by the present disclosure is to use multiple lane lines (lane lines are also called 69 lines) that remain stationary and have relatively stable positional relationships in the traffic scene as a reference, and by analyzing the pixel coordinate conversion relationship (referred to as the second coordinate conversion matrix in this article, which also belongs to the homography matrix) between the contours of multiple lane lines in the first original image and the contours of multiple lane lines in the second original image, to clarify the degree of correction required for the first coordinate conversion matrix.

[0084] Considering the shooting range of the camera, the two ends of the road surface extend infinitely in the driving direction. Therefore, it is difficult to distinguish between different sections of the road surface, and it is difficult to clarify which section contour in the first original image matches which section contour in the second original image, and it is also difficult to calculate the pixel coordinate conversion relationship between the images obtained in different shooting poses based on the two matching section contours.

[0085] Figure 3 An exemplary process of a method for correcting the coordinate conversion matrix is provided, including the following steps:

[0086] S300: After determining that the current pose of the camera is different from the preset pose, based on the first original image, obtain a third contour image highlighting multiple continuous lane line contours on the road surface; based on the second original image, obtain a fourth contour image highlighting multiple continuous lane line contours on the road surface.

[0087] It is easy to understand that if there are multiple second original images, any one of the second original images can be selected to execute step S300.

[0088] S302: Collect M third pixel points from the boundaries of multiple continuous lane line contours in the third contour image, and collect M fourth pixel points from the boundaries of multiple continuous lane line contours in the fourth contour image.

[0089] It should be noted that the M third pixel points and the M fourth pixel points correspond to each other one by one. The relative position of any first pixel point on the boundary of the lane line contour in the first contour image is the same as the relative position of its corresponding second pixel point on the boundary of the lane line contour in the second contour image; M > 1.

[0090] S304: Determine the second coordinate transformation matrix according to the pixel coordinate transformation relationship between each group of corresponding third pixel points and fourth pixel points.

[0091] S306: Correct the first coordinate transformation matrix based on the second coordinate transformation matrix.

[0092] Through step S306, the corrected first coordinate transformation matrix is used to convert the pixel point coordinates in the image obtained by the camera shooting the traffic scene in the current pose into longitude and latitude coordinates. Usually, the first coordinate transformation matrix and the second coordinate transformation matrix can be multiplied to obtain the corrected first coordinate transformation matrix.

[0093] The method for determining that the current pose of the camera is different from the preset position can be Figure 1 the method shown, or other methods in the prior art.

[0094] In some embodiments, the road surface may have double lanes with opposite driving directions. In this case, multiple consecutive lane line contours in the third contour image and multiple consecutive lane line contours in the fourth contour image should generally be lane line contours on the same lane.

[0095] In some embodiments, considering that in the traffic scene, the line types on the road surface include not only lane lines but also other lines, such as U-turn lines. For this reason, the lane lines can be formally defined as dotted lines. Based on the first original image, a fifth contour image highlighting the contours of the dotted line type targets in the road surface can be obtained; based on the edge detection algorithm, the contour of each segment in the contour of the dotted line type targets in the fifth contour image is detected to obtain a third contour image highlighting multiple consecutive lane line contours in the road surface; where each segment represents a lane line, and the contour of each segment is used as a lane line contour.

[0096] Furthermore, if the shape of the lane line contour detected based on the edge detection algorithm is irregular, the lane line contour can be rectangularly fitted to obtain a lane line contour close to a rectangular shape.

[0097] In practical applications, it is usually necessary to ensure that each lane line contour in the third contour image corresponds one-to-one with each lane line contour in the fourth contour image. Two lane line contours that have a corresponding relationship and are included in different images are usually the contours of the same lane line in the real scene in different images, that is, they are the different lane line contours obtained by the camera shooting the same lane line in different poses.

[0098] This means that the following alignment conditions need to be met: ensure that the number of each continuous lane line contour in the third contour image is the same as the number of each continuous lane line contour in the fourth contour image; and, it is necessary to ensure that the spacing between adjacent two lane line contours in the third contour image and the fourth contour image generally conforms to the conventional lane line spacing (for example, 9 meters); and, it is necessary to ensure that each lane line contour in the third contour image and the fourth contour image is the contour of a complete lane line, rather than the contour of a half lane line.

[0099] When the above alignment conditions are met, it can be ensured that each lane line contour in the third contour image corresponds one-to-one with each lane line contour in the fourth contour image.

[0100] If the above alignment conditions are not met, it is necessary to first perform alignment processing on each lane line contour in the third contour image and each lane line contour in the fourth contour image to meet the above alignment conditions. For example, delete the mismatched lane line contours.

[0101] The present disclosure also provides a computer-readable storage medium, as Figure 4 shown, a computer program is stored on the medium 140, and when the program is executed by a processor, the method of the embodiment of the present disclosure is implemented.

[0102] The present disclosure also provides a computing device, including a memory and a processor; the memory is used to store computer instructions that can run on the processor, and the processor is used to implement the method of the embodiment of the present disclosure when executing the computer instructions.

[0103] Figure 5 is a schematic structural diagram of a computing device provided by the present disclosure. The computing device 15 may include, but is not limited to: a processor 151, a memory 152, and a bus 153 connecting different system components (including the memory 152 and the processor 151).

[0104] Among them, the memory 152 stores computer instructions that can be executed by the processor 151, enabling the processor 151 to execute the methods of any embodiment of the present disclosure. The memory 152 may include a random access storage unit RAM 1521, a cache storage unit 1522, and / or a read-only storage unit ROM 1523. The memory 152 may further include: a program tool 1525 having a set of program modules 1524, and the program modules 1524 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, and one or more combinations of these program modules may include the implementation of a network environment.

[0105] The bus 153 may include, for example, a data bus, an address bus, and a control bus, etc. The computing device 15 may also communicate with an external device 155 through the I / O interface 154, and the external device 155 may be, for example, a keyboard, a Bluetooth device, etc. The computing device 150 may also communicate with one or more networks through the network adapter 156. For example, the network may be a local area network, a wide area network, a public network, etc. As shown in the figure, the network adapter 156 may also communicate with other modules of the computing device 15 through the bus 153.

[0106] In addition, although the operations of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0107] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefit. This division is only for the convenience of expression. The present disclosure aims to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0108] The systems, devices, modules, or units illustrated in the above embodiments may be specifically implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0109] For the convenience of description, when describing the above device, it is divided into various units according to functions for separate description. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0110] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0111] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one or more of the processes Figure 1 or multiple processes and / or blocks

[0112] This specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0113] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the specified functions in Figure 1 one or more of the processes Figure 1 or multiple processes and / or blocks

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing in the process Figure 1 one process or multiple processes and / or blocks Figure 1 steps of the functions specified in one block or multiple blocks. In a typical configuration, a computer includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0115] The memory may include non-permanent memory in the computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0116] Computer-readable media includes permanent and non-permanent, removable and non-removable media and can be implemented by any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0117] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, commodity or device comprising the element.

[0118] The above describes multiple embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0119] The terms used in multiple embodiments of this specification are for the purpose of describing particular embodiments only and are not intended to limit multiple embodiments of this specification. The singular forms "a", "the", and "said" used in multiple embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0120] It should be understood that although the terms first, second, third, etc. may be used in multiple embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of multiple embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "upon" or "in response to determining".

[0121] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for method embodiments, since they are basically similar to method embodiments, they are described relatively simply, and the relevant parts can be referred to the description of the method embodiments. The method embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated. When implementing the solutions of the embodiments of this specification, the functions of the various modules can be implemented in one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the solutions of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0122] The above is only the preferred embodiment of multiple embodiments of this specification and is not intended to limit multiple embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of multiple embodiments of this specification shall be included within the scope of protection of multiple embodiments of this specification.

Claims

1. A method for detecting changes in the shooting pose of a camera, including: obtaining a first original image captured by the camera of the real scene in a preset pose, and obtaining a second original image captured by the camera of the real scene in the current pose; wherein, the real scene includes a reference object that remains stationary, and the camera is stationary relative to the reference object; based on the first original image, obtaining a first contour image highlighting the contour of the reference object; based on the second original image, obtaining a second contour image highlighting the contour of the reference object; analyzing the degree of offset of the reference object contour in the second contour image relative to the reference object contour in the first contour image as the offset degree corresponding to the second contour image; if the offset degree corresponding to the second contour image exceeds a first preset degree, determining that the current pose is different from the preset pose.

2. The method according to claim 1, the step of analyzing the degree of offset of the reference object contour in the second contour image relative to the reference object contour in the first contour image, including: performing an intersection over union (IOU) calculation on the reference object contour in the second contour image and the reference object contour in the first contour image; in the case where the IOU calculation result is greater than a first ratio, determining that the offset degree corresponding to the second contour image is lower than or equal to the preset degree.

3. The method according to claim 2, in the case where the IOU calculation result is less than or equal to the first ratio, the step of analyzing the degree of offset of the reference object contour in the second contour image relative to the reference object contour in the first contour image further includes: analyzing the degree of overlap between the boundary of the reference object contour in the second contour image and the boundary of the reference object contour in the first contour image, and if the degree of overlap is lower than a second preset degree, determining that the offset degree corresponding to the second contour image exceeds the first preset degree.

4. The method according to claim 3, analyzing the degree of overlap between the boundary of the reference object contour in the second contour image and the boundary of the reference object contour in the first contour image, including: sampling N second pixel points from the boundary of the reference object contour in the second contour image, and sampling N first pixel points from the boundary of the reference object contour in the first contour image; wherein, the N second pixel points and the N first pixel points correspond one by one, and the first pixel point and the second pixel point with a corresponding relationship form a pixel point combination, and the relative position of the first pixel point on the boundary of the reference object contour in the first contour image in the same pixel point combination is the same as the relative position of the corresponding second pixel point on the boundary of the reference object contour in the second contour image; N>1; calculating the relative distance between the first pixel point and the second pixel point in each pixel point combination in the image coordinate system, and determining the proportion of the pixel point combinations with a relative distance less than the preset distance in all pixel point combinations as the degree of overlap between the boundary of the reference object contour in the second contour image and the boundary of the reference object contour in the first contour image.

5. The method according to claim 3, the step of analyzing the offset degree of the reference object contour in the second contour image relative to the reference object contour in the first contour image, further comprises: If the overlapping degree is higher than a third preset degree, it is determined that the offset degree corresponding to the second contour image is lower than or equal to a first preset degree; wherein, the third preset degree is greater than the second preset degree.

6. The method according to claim 5, wherein, the real scene further includes a set of background objects that remain stationary; the step of analyzing the offset degree of the reference object contour in the second contour image relative to the reference object contour in the first contour image further comprises: If the overlapping degree is higher than or equal to the second preset degree and lower than or equal to the third preset degree, obtain a first background image highlighting the set of background objects obtained based on the first original image, and a second background image highlighting the set of background objects corresponding to the second original image corresponding to the second contour image; Calculate the similarity of the pixel point distribution structures of the first background image and the second background image as the similarity corresponding to the second contour image; If the similarity corresponding to the second background image exceeds the specified similarity, it is determined that the offset degree corresponding to the second contour image is lower than or equal to the first preset degree; If the similarity corresponding to the second background image is lower than or equal to the specified similarity, it is determined that the offset degree corresponding to the second contour image exceeds the first preset degree.

7. The method according to any one of claims 1-6, further comprises: After determining that the current pose of the camera is different from the preset pose, calculate the current pose according to the first original image, the second original image and the preset pose, so as to calculate real scene related data based on the current pose.

8. The method according to claim 1, further comprising After determining that the current pose is different from the preset pose, obtain a third original image captured by the camera of the real scene in the current pose; Based on the third original image, obtain a corresponding third contour image highlighting the reference object contour; Analyze the offset degree of the reference object contour in the third contour image relative to the reference object contour in the first contour image as the offset degree corresponding to the third contour image; If the offset degree corresponding to the third contour image exceeds the first preset degree, it is determined that the current pose has returned to the preset pose.

9. The method according to claim 1, wherein, the number of the second original images is at least two; If the offset degree corresponding to the second contour image exceeds the first preset degree, determining that the current pose is different from the preset pose includes: If the offset degree corresponding to each second contour image exceeds the first preset degree, it is determined that the current pose is different from the preset pose.

10. The method according to any one of claims 1-6, 8-9, wherein, the real scene is a traffic scene and the reference object is a road surface.

11. A method for correcting a coordinate transformation matrix, wherein, The first coordinate transformation matrix is used to transform the pixel point coordinates in the image obtained by the camera shooting the traffic scene in the preset pose into longitude and latitude coordinates; the road surface in the traffic scene includes a plurality of lane lines arranged continuously at a fixed interval, and the method includes: After determining that the current pose of the camera is different from the preset pose based on the method described in claim 9, based on the first original image, a third contour image highlighting a plurality of continuous lane line contours on the road surface is obtained; based on the second original image, a fourth contour image highlighting a plurality of continuous lane line contours on the road surface is obtained; Collect M third pixel points from the boundaries of a plurality of continuous lane line contours in the third contour image, and collect M fourth pixel points from the boundaries of a plurality of continuous lane line contours in the fourth contour image; wherein, the M third pixel points correspond to the M fourth pixel points one by one, and the relative position of any first pixel point on the boundary of the lane line contour in the first contour image is the same as the relative position of its corresponding second pixel point on the boundary of the lane line contour in the second contour image; M>1; Determine the second coordinate transformation matrix according to the pixel coordinate transformation relationship between each group of corresponding third pixel points and fourth pixel points; Based on the second coordinate transformation matrix, correct the first coordinate transformation matrix so that the corrected first coordinate transformation matrix is used to transform the pixel point coordinates in the image obtained by the camera shooting the traffic scene in the current pose into longitude and latitude coordinates.

12. The method according to claim 11, wherein the road surface has a two-lane with opposite driving directions; The plurality of continuous lane line contours in the third contour image and the plurality of continuous lane line contours in the fourth contour image are lane line contours on the same lane.

13. The method according to claim 11, based on the first original image, obtaining a third contour image highlighting a plurality of continuous lane line contours on the road surface, including: Based on the first original image, obtain a fifth contour image highlighting the contour of the target of the dotted line type on the road surface; Based on the edge detection algorithm, detect the contour of each line segment in the contour of the target of the dotted line type in the fifth contour image, Obtain a third contour image highlighting a plurality of continuous lane line contours on the road surface; wherein, each line segment represents a lane line, and the contour of each line segment is used as a lane line contour.

14. A computing device, including a memory and a processor; the memory is used to store computer instructions that can be run on the processor, and the processor is used to implement the method according to any one of claims 1 to 13 when executing the computer instructions.

15. A computer-readable storage medium, on which a computer program is stored, and the program implements the method according to any one of claims 1 to 13 when executed by a processor.

Images