Image correction method and device, storage medium and electronic equipment

Abstract

The invention discloses an image correction method and device, a storage medium and electronic equipment. The method relates to the field of artificial intelligence and big data, and comprises the following steps: obtaining a target image acquired by a target camera at a target moment and a reference image acquired at a reference moment, the target moment being later than the reference moment; according to an image offset scene between the target image and the reference image, a target offset correction mode for the target camera is determined, and the image offset scene is one of a scene generating image offset along with time accumulation and a scene generating image offset due to weather factors or human factors; determining offset correction information based on the target offset correction mode, the target image and the reference image; and performing image correction on the target image based on the offset correction information. Through the method and the device, the problem of low correction accuracy when the image shot by the camera is corrected under the condition that the pose of the camera is changed in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and big data, in particular, to an image correction method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the continuous development of camera technology, high tower cameras, as an advanced monitoring device, have gradually become one of the important tools in the fields of safety monitoring, engineering construction monitoring and traffic supervision. When monitoring, relevant personnel often manually mark or identify algorithms on the monitoring screen to mark some positions or enclose an electronic fence, so that the relevant personnel can better focus on the corresponding monitoring position or monitoring area when adjusting the pose or focal length of the high tower camera. However, due to the erection environment of the high tower camera, the position and shooting angle of the high tower camera may change over time, and in coastal areas, the camera will continue to shake due to strong winds, so that the marked position and the electronic fence on the monitoring screen are offset, hindering the normal function of the high tower camera.

[0003] At present, in the case that the pose of the camera changes, the related technology usually relies on relevant personnel to correct the image shot by the camera, thereby having the problem of low correction accuracy.

[0004] In view of the above problems in the related art, no effective solution has been proposed so far. SUMMARY

[0005] The main purpose of the present application is to provide an image correction method, device, storage medium and electronic device to solve the problem of low correction accuracy when correcting the image shot by the camera in the case that the pose of the camera changes.

[0006] In order to achieve the above purpose, according to one aspect of the present application, an image correction method is provided. The method comprises: acquiring a target image collected by a target camera at a target time and a reference image collected at a reference time, wherein the target time is later than the reference time; determining a target offset correction mode for the target camera according to an image offset scenario between the target image and the reference image, wherein the image offset scenario is one of the following: a scenario of image offset caused by time accumulation, a scenario of image offset caused by weather factors or human factors; determining offset correction information based on the target offset correction mode, the target image and the reference image; and performing image correction on the target image based on the offset correction information.

[0007] Further, the image correction method further comprises: in the case that the image offset scenario is the scenario that the image offset is caused by weather factors or human factors, determining the target offset correction manner for the target camera before receiving a scenario trigger instruction of a user; in the case that the scenario trigger instruction is not received and the target time is a preset time, determining the image offset scenario as the scenario that the image offset is accumulated over time.

[0008] Further, in the case that the image offset scenario is the scenario that the image offset is accumulated over time, the target offset correction manner refers to a manner of performing offset correction based on mutual information, and the image correction method further comprises: obtaining camera parameters of the target camera, wherein the camera parameters at least include distortion parameters; performing distortion correction on the reference image and the target image based on the camera parameters to obtain a first reference image and a first target image; constructing a homography matrix based on an initial offset parameter, and performing image mapping on the first reference image based on the homography matrix to obtain a second reference image; determining an offset parameter of the target image relative to the reference image based on a first target function and mutual information between the second reference image and the first target image, and determining offset correction information based on the offset parameter, wherein the first target function aims to minimize the reciprocal of the mutual information.

[0009] Further, the image correction method further comprises: adjusting the pose of the target camera based on the offset parameter; and performing image acquisition based on the target camera after the pose is adjusted to obtain an updated target image, so as to complete the image correction.

[0010] Further, in the case that the image offset scenario is the scenario that the image offset is caused by weather factors or human factors, the target offset correction manner refers to a manner of performing offset correction based on a corner point matching algorithm, and the image correction method further comprises: obtaining camera parameters of the target camera, wherein the camera parameters at least include distortion parameters; performing distortion correction on the reference image and the target image based on the camera parameters to obtain a first reference image and a first target image; matching corner points in the first reference image and the first target image based on a corner point detection algorithm to obtain a plurality of corner point pairs; determining a target homography matrix of the target image relative to the reference image based on the plurality of corner point pairs, and determining offset correction information based on the target homography matrix.

[0011] Further, the reference image includes annotation information, and the annotation information is used to identify a region of interest in the reference image, and the image correction method further comprises: mapping the annotation information in the reference image to the target image based on the target homography matrix to obtain an updated target image, so as to complete the image correction.

[0012] Further, the image correction method further comprises: determining a shooting scene of the target camera, wherein the shooting scene is one of: a scene in which the obtained image contains M three-dimensional straight lines, a scene in which the obtained image contains N three-dimensional straight lines and the target camera is rotatable, and a scene in which the obtained image contains N three-dimensional straight lines and the target camera is not rotatable, M is greater than a preset value, and N is less than or equal to the preset value; in a case where the shooting scene is the scene in which the obtained image contains M three-dimensional straight lines, determining the camera parameter based on the target Hough transform method; in a case where the shooting scene is the scene in which the obtained image contains N three-dimensional straight lines and the target camera is rotatable, rotating the target camera, and determining the camera parameter based on the collected image before the target camera is rotated and the collected image after the target camera is rotated; and in a case where the shooting scene is the scene in which the obtained image contains N three-dimensional straight lines and the target camera is not rotatable, erecting a plurality of three-dimensional target points in a layout site of the target camera, and determining the camera parameter based on a plurality of target point images obtained by the target camera shooting the plurality of three-dimensional target points.

[0013] To achieve the above object, according to another aspect of the present application, an image correction device is provided. The device comprises: an acquisition module configured to acquire a target image collected by a target camera at a target time and a reference image collected by the target camera at a reference time, wherein the target time is later than the reference time; a first determination module configured to determine a target offset correction mode for the target camera according to an image offset scene between the target image and the reference image, wherein the image offset scene is one of: a scene in which image offset is accumulated over time, and a scene in which image offset is caused by weather factors or human factors; a second determination module configured to determine offset correction information based on the target offset correction mode, the target image and the reference image; and a correction module configured to perform image correction on the target image based on the offset correction information.

[0014] Further, the image correction device further comprises: a third determination module configured to determine that the image offset scene is the scene in which image offset is caused by weather factors or human factors in a case where a scene triggering instruction of a user is received; and a fourth determination module configured to determine that the image offset scene is the scene in which image offset is accumulated over time in a case where the scene triggering instruction is not received and the target time is a preset time.

[0015] Further, the second determining module further comprises: a first obtaining sub-module, configured to obtain camera parameters of the target camera, wherein the camera parameters at least comprise distortion parameters; a first correcting sub-module, configured to perform distortion correction on the reference image and the target image respectively based on the camera parameters, to obtain a first reference image and a first target image; a first mapping sub-module, configured to construct a homography matrix based on the initial offset parameters, and perform image mapping on the first reference image based on the homography matrix, to obtain a second reference image; and a first determining sub-module, configured to determine offset parameters of the target image relative to the reference image based on a first target function and mutual information between the second reference image and the first target image, and determine offset correction information based on the offset parameters, wherein the first target function aims to minimize the reciprocal of the mutual information.

[0016] Further, the correcting module further comprises: an adjusting sub-module, configured to adjust the pose of the target camera based on the offset parameters; and an acquiring sub-module, configured to perform image acquisition based on the target camera after the pose adjustment, to obtain an updated target image, so as to complete the image correction.

[0017] Further, the second determining module further comprises: a second obtaining sub-module, configured to obtain camera parameters of the target camera, wherein the camera parameters at least comprise distortion parameters; a second correcting sub-module, configured to perform distortion correction on the reference image and the target image respectively based on the camera parameters, to obtain a first reference image and a first target image; a matching sub-module, configured to match the corner points in the first reference image and the first target image based on a corner detection algorithm, to obtain a plurality of corner point pairs; and a second determining sub-module, configured to determine a target homography matrix of the target image relative to the reference image based on the plurality of corner point pairs, and determine offset correction information based on the target homography matrix.

[0018] Further, the correcting module further comprises: a second mapping sub-module, configured to map the annotation information in the reference image to the target image based on the target homography matrix, to obtain an updated target image, so as to complete the image correction.

[0019] Further, the first obtaining sub-module or the second obtaining sub-module further comprises: a third determining sub-module, configured to determine a shooting scene of the target camera, wherein the shooting scene is one of: a scene in which the image obtained by shooting contains M three-dimensional straight lines, a scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, and a scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is not rotatable, M is greater than a preset value, and N is less than or equal to the preset value; a fourth determining sub-module, configured to determine the camera parameter based on a target Hough transform method in a case where the shooting scene is the scene in which the image obtained by shooting contains M three-dimensional straight lines; a fifth determining sub-module, configured to rotate the target camera in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, and determine the camera parameter based on the collected image before the rotation of the target camera and the collected image after the rotation of the target camera; and a sixth determining sub-module, configured to erect a plurality of three-dimensional target points in a layout site of the target camera in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is not rotatable, and determine the camera parameter based on a plurality of target point images obtained by the target camera shooting the plurality of three-dimensional target points.

[0020] To achieve the above object, according to another aspect of the present application, a computer readable storage medium is provided, which comprises a stored executable program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the image correction method described above when the executable program is run.

[0021] To achieve the above object, according to another aspect of the present application, an electronic device is provided, which comprises a memory storing an executable program, and a processor configured to run the program, wherein the program is configured to execute the image correction method described above when the program is run.

[0022] To achieve the above object, according to another aspect of the present application, a computer program product is provided, which comprises computer instructions configured to implement the steps of the image correction method described above when executed by a processor.

[0023] In the embodiment of the present application, the target image collected by the target camera at the target time and the reference image collected at the reference time are obtained, so as to effectively obtain the images collected before and after the change of the target camera pose. The target offset correction mode for the target camera is determined according to the image offset scene between the target image and the reference image, so as to determine the offset correction mode based on the scene causing the image offset between the target image and the reference image. Therefore, when the offset correction information is determined based on the target offset correction mode, the target image and the reference image, the offset correction information is determined based on the difference between the images collected before and after the change of the target camera pose and the specific correction mode, so as to effectively improve the correction accuracy during image correction.

[0024] Therefore, the method provided in the present application achieves the purpose of image correction by using the offset correction mode corresponding to the image offset scene, and achieves the technical effect of improving the correction accuracy. The technical problem of low correction accuracy when correcting the images captured by the camera in the case of the change of the camera pose is solved. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and the illustrative embodiments of the present application and their description serve the purpose of explaining the present application. The accompanying drawings should not be construed as an inappropriate limitation on the present application. In the drawings:

[0026] Figure 1 is a hardware structure block diagram of a computer terminal according to an embodiment of the present application;

[0027] Figure 2 is a flowchart of an image correction method according to an embodiment of the present application;

[0028] Figure 3 is a flowchart of determining offset correction information according to an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of an image correction device according to an embodiment of the present application;

[0030] Figure 5 is a structure block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present application are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal. For example, the system and related users or institutions are provided with an interface to provide the user with a corresponding operation portal for the user to choose to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0034] Embodiment 1

[0035] According to the embodiments of the present application, an embodiment of an image correction method is also provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0036] The method embodiment provided by the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an image correction method is shown. As shown inFigure 1 As shown, the computer terminal 10 (or mobile device) can include one or more processors 102 (the processor 102 can include, but not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can also include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can also include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can also include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any combination thereof. In addition, the data processing circuit can be a single independent processing module, or any one of the other elements combined into the computer terminal 10 (or mobile device) in whole or in part. As referred to in the embodiments of the present application, the data processing circuit serves as a processor to control (for example, selection of a variable resistance terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the image correction method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned image correction method. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 can further include a memory remotely disposed with respect to the processor 102, which can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0040] The display can be a liquid crystal display (LCD) that is touch screen type, for example, which can enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0041] In the above operating environment, the present application provides an image correction method as shown in the following. Figure 2 Figure 2 is a flowchart of the image correction method according to an embodiment of the present application.

[0042] In step S201, a target image captured by a target camera at a target time and a reference image captured by the target camera at a reference time are obtained, wherein the target time is later than the reference time.

[0043] Optionally, an electronic device, an application system, a server, or the like can be taken as an execution subject of the present application. In the present embodiment, a target processing system is taken as the execution subject to execute the above image correction method.

[0044] Optionally, the target camera can be a camera arranged at a high place, such as a high tower camera, i.e., a camera arranged on a high tower. The target image can refer to an image captured by the target camera at a certain current time (i.e., the target time). The reference image can refer to an image captured by the target camera at a previously set time point (i.e., the reference time), which is a time point at which the camera pose and parameters are considered to be stable and known, and is a reference for later comparison and correction. For example, the time point can be a time point at which the target camera is just arranged or just adjusted. The target time is later than the reference time, and the target time is a time point at which the camera pose can have changed, for example, a time point after a period of operation, weather change, or human operation.

[0045] In step S202, a target offset correction mode for the target camera is determined according to an image offset scenario between the target image and the reference image, wherein the image offset scenario is one of the following: a scenario in which the image offset is accumulated over time, a scenario in which the image offset is caused by weather factors or human factors.

[0046] ​Optionally, the image offset scenario refers to a scenario causing image offset between the target image and the reference image. For example, a stable offset of the camera caused by long-term accumulation of external factors, which is equivalent to a scenario of accumulating image offset over time, a sustained unstable offset of the camera caused by weather conditions such as strong wind, or a scenario of image offset caused by weather factors or human factors caused by the active adjustment of the monitoring personnel to the monitoring area.

[0047] Optionally, the image offset scenario between the target image and the reference image can be input to the target processing system after being determined by a human.

[0048] Optionally, the image offset scenario between the target image and the reference image can also be determined by the target processing system, for example, if it is detected that the wind level of the area where the target camera is located is greater than a preset level, or if it is detected that the user adjusts the annotation information in the target camera shooting picture, it is determined that the image offset scenario is a scenario of image offset caused by weather factors or human factors, otherwise, if the foregoing conditions are not detected and the target time is a preset time, it is determined that the image offset scenario is a scenario of accumulating image offset over time. The annotation information in the target camera shooting picture is used to identify the region of interest in the target camera shooting picture.

[0049] Optionally, different image offset scenarios are matched with different offset correction methods, and the target processing system can be preconfigured with a matching relationship between different image offset scenarios and different offset correction methods, so that after determining the image offset scenario between the target image and the reference image, the target offset correction method is determined from the plurality of offset correction methods based on the matching relationship.

[0050] Step S203, based on the target offset correction method, the target image and the reference image, determine the offset correction information.

[0051] Optionally, in the case of the image offset scenario being a scenario of accumulating image offset over time, the target offset correction method refers to a method of performing offset correction based on mutual information. The target processing system can determine the offset parameter of the target image relative to the reference image based on the method, the target image and the reference image, and thus determine the offset correction information based on the offset parameter.

[0052] Optionally, in the case of the image offset scenario being a scenario of image offset caused by weather factors or human factors, the target offset correction method refers to a method of performing offset correction based on a corner matching algorithm. The target processing system can determine the target homography matrix of the target image relative to the reference image based on the method, the target image and the reference image, and thus determine the offset correction information based on the target homography matrix.

[0053] Step S204, based on the offset correction information, perform image correction on the target image.

[0054] Optionally, after determining the offset correction information, the target processing system can adjust the gimbal position of the target camera or update the annotation information on the target image based on the offset correction information, so that the target image can be aligned or kept consistent with the reference image.

[0055] In the embodiments of the present application, by acquiring the target image collected by the target camera at the target time and the reference image collected at the reference time, the effective acquisition of the images collected before and after the change of the target camera pose is realized. By determining the target offset correction mode for the target camera according to the image offset scene between the target image and the reference image, the offset correction mode is determined based on the scene causing the image offset between the target image and the reference image, so that when the offset correction information is determined based on the target offset correction mode, the target image and the reference image, the offset correction information is determined based on the difference between the images collected before and after the change of the target camera pose and the specific correction mode, which can effectively improve the correction accuracy during image correction.

[0056] Therefore, the method provided by the present application achieves the purpose of image correction using the offset correction mode corresponding to the image offset scene, realizes the technical effect of improving the correction accuracy, and solves the technical problem of low correction accuracy when the camera pose changes in the related art.

[0057] Optionally, in the image correction method provided by the embodiments of the present application, before determining the target offset correction mode for the target camera according to the image offset scene between the target image and the reference image, the method further includes: in the case that the scene trigger instruction of the user is received, determining that the image offset scene is the scene of image offset caused by weather factors or human factors; in the case that the scene trigger instruction is not received and the target time is a preset time, determining that the image offset scene is the scene of image offset accumulated over time.

[0058] Optionally, when the image offset is caused by weather factors or human factors, such as the case that the target camera is arranged in an area with strong wind or the annotation information of the camera collection picture is adjusted by human, the user can send a scene trigger instruction to the target processing system. The target processing system determines that the image offset scene is the scene of image offset caused by weather factors or human factors in the case that the scene trigger instruction is received. The scene trigger instruction is used to represent that the image offset scene between the target image and the reference image is the scene of image offset caused by weather factors or human factors.

[0059] Optionally, the target camera can generate a stable offset under long-term accumulation of external force factors, and thus the target processing system can periodically perform image correction to overcome the pose change of the target camera caused by time factors. Therefore, the preset time point can be multiple, and the time interval between the multiple preset time points is the same. In a case where the scene trigger instruction is not received and the target time point is a preset time point, the target processing system determines that the time point for periodically performing image correction is reached, so as to determine that the image offset scene is a scene in which the image offset is generated due to time accumulation.

[0060] Optionally, in a case where the scene trigger instruction is received and the target time point is a preset time point, it is determined that the image offset scene is a scene in which the image offset is caused by weather factors or human factors.

[0061] Optionally, in a case where the scene trigger instruction is not received and the target time point is not a preset time point, it is determined that the image offset scene is not triggered, that is, the target image does not need to be corrected.

[0062] It should be noted that, by the above manner, the image offset scene between the target image and the reference image is accurately determined, so that the accuracy of determining the target offset correction manner can be improved, and thus the image correction accuracy can be improved.

[0063] Optionally, in the image correction method provided in the embodiments of the present application, in a case where the image offset scene is a scene in which the image offset is generated due to time accumulation, the target offset correction manner is a manner of performing offset correction based on mutual information, wherein, Figure 3 is a flowchart for determining offset correction information according to the embodiments of the present application, as shown in Figure 3 In the process of determining the offset correction information based on the target offset correction manner, the target image and the reference image, the target processing system can perform the following steps:

[0064] Step S301: Obtain camera parameters of the target camera, wherein the camera parameters at least include distortion parameters.

[0065] Optionally, the camera parameters at least include distortion parameters, and can also include focal length (lateral focal length, longitudinal focal length) and optical center coordinates (lateral offset, longitudinal offset), wherein the distortion parameters are used to correct lens distortion, the focal length represents the projection length of the focal length of the camera lens on the image plane, and the optical center coordinates represent the position of the origin of the image coordinate system in the pixel coordinate system.

[0066] Optionally, the camera parameters of the target camera can be determined by a person in advance, or can be determined by the target processing system based on the image captured by the camera.

[0067] At step S302, the reference image and the target image are respectively rectified based on the camera parameters to obtain a first reference image and a first target image.

[0068] After the camera parameters of the target camera are obtained, the target processing system can rectify the reference image and the target image based on the camera parameters to eliminate the distortion effect in the images, to obtain a first reference image and a first target image.

[0069] At step S303, a homography matrix is constructed based on the initialization offset parameters, and the first reference image is mapped based on the homography matrix to obtain a second reference image.

[0070] Optionally, the initialization offset parameters can include at least one of a rotation parameter, a translation parameter, and a scaling parameter, and the parameter values of the parameters in the initialization offset parameters are preset initial values. After the first reference image and the first target image are obtained, the target processing system can construct a homography matrix based on the initialization offset parameters. The homography matrix is a 3x3 matrix used to describe the geometric transformation between two images.

[0071] After the homography matrix is determined, the target processing system can map the points in the first reference image to the coordinate system of the first target image using the homography matrix, to obtain a second reference image.

[0072] At step S304, the offset parameters of the target image relative to the reference image are determined based on a first target function and the mutual information between the second reference image and the first target image, and the offset correction information is determined based on the offset parameters, wherein the first target function aims to minimize the reciprocal of the mutual information.

[0073] Optionally, the mutual information between the second reference image and the first target image is calculated. The mutual information is a method for measuring the similarity between two images, and the greater the mutual information, the higher the similarity between the two images.

[0074] For example, the histograms H1 and H2 of the second reference image and the first target image are respectively calculated, and the joint histogram H 12 between the second reference image and the first target image is calculated. 12 Then, the histograms of the second reference image and the first target image and the joint histogram are normalized to probability distributions p1, p2, and p 12 The mutual information between the second reference image and the first target image is calculated based on the probability distributions p1, p2, and p

[0075] Optionally, the reciprocal of the mutual information is defined as a mutual information loss, and a first objective function is defined to minimize the mutual information loss. Thus, after the mutual information between the first target images is determined, the function value of the first objective function is determined based on the mutual information, and in a case where the function value is greater than a preset first threshold, the initialization offset parameter is adjusted using an optimization algorithm (such as a gradient descent method), and then the steps of constructing a new homography matrix based on the adjusted initialization offset parameter, performing image mapping on the first reference image based on the new homography matrix to obtain an updated second reference image, and calculating the function value of the first objective function based on the first objective function and the mutual information between the updated second reference image and the first target image are repeatedly performed until the function value of the first objective function is less than or equal to the preset first threshold, and in this case, the current updated initialization offset parameter is determined as the offset parameter of the target image relative to the reference image.

[0076] Optionally, the offset parameter of the target image relative to the reference image is used to describe the geometric transformation relationship of the target image relative to the reference image. After the offset parameter is determined, the offset parameter can be determined as offset correction information.

[0077] In another optional embodiment, in a case where the function value of the first objective function is less than or equal to the preset first threshold, the current homography matrix can also be decomposed under the homography assumption to obtain a corresponding rotation matrix and a camera parameter matrix corresponding to the target image, then the rotation matrix is converted from the three-dimensional coordinate system of the camera to the rotation matrix in the control gimbal coordinate system according to the pitch angle when the reference image is shot, and then the rotation matrix is further decomposed to obtain specific offset parameters, and the relative offset of the camera position is decomposed according to the offset parameters and the camera parameter matrix corresponding to the offset image, and then the relative offset of the camera position is determined as the offset correction information. Wherein, the foregoing control gimbal is used to control the target camera to move.

[0078] It should be noted that by minimizing the reciprocal of the mutual information as the optimization objective, the slight deviation of the target camera pose caused by time accumulation can be effectively addressed, thereby improving the accuracy of determining the offset correction information.

[0079] Optionally, in the image correction method provided in the embodiments of the present application, the target image is corrected based on the offset correction information, including: adjusting the pose of the target camera based on the offset parameter; and performing image acquisition based on the target camera after the pose is adjusted to obtain an updated target image, so as to complete the image correction.

[0080] Since the offset parameter of the target image relative to the reference image is used to describe the geometric transformation relationship of the target image relative to the reference image, the target processing system can generate an adjustment instruction for the gimbal based on the offset parameter, to compensate for the pose offset of the target camera based on the adjustment instruction, where the gimbal is also called a gimbal mechanism or a gimbal base, and is used to carry and drive the target camera to move (i.e., adjust the pose). For example, if the offset parameter indicates that the target camera has a 3-degree rotation offset on the vertical axis and a 5-millimeter translation offset in the horizontal direction, the adjustment instruction can be used to control the gimbal to rotate reversely by 3 degrees and reversely translate by 5 millimeters in the horizontal direction, to compensate for the rotation offset and the translation offset of the target camera.

[0081] Optionally, after the pose adjustment of the target camera is completed, the target processing system can control the target camera to perform image acquisition again, at this time, the image acquired will exhibit a higher degree of similarity with the reference image due to the correction of the camera pose, and will not be affected by the long-term accumulated pose offset. The target processing system can take the newly acquired image as an updated target image, and replace the target image with the updated target image for storage and video monitoring activities, so as to realize image correction.

[0082] In an optional embodiment, if the function value of the first objective function is still greater than the preset first threshold after N iterations in the process of iteratively determining the offset parameter based on mutual information, in this case, the iteration is stopped, and an alarm event is issued to remind the user (such as a video monitoring personnel) to correct the annotation position by manual correction or to correct the camera pose by manual tower climbing.

[0083] Optionally, when the monitoring personnel actively adjusts the monitoring by rotating the camera, the present application supports image stitching and annotation position updating in different coordinate systems, such as a cylindrical coordinate system: the content of each image is mapped to a cylindrical surface around the camera, the axis of the cylinder passes through the target camera and is parallel to the vertical coordinate axis of the camera coordinate system, each point on the image is restored to the corresponding point on the cylindrical surface, and finally the cylindrical surface is unfolded, and the stitched cylindrical image is obtained. The corresponding annotation position is updated according to the geometric relationship.

[0084] It should be noted that, in the scenario where the image offset is accumulated over time, the deviation of the image is caused by the accumulated offset of the camera pose, and therefore, in this scenario, adjusting the camera pose based on the offset correction information can effectively offset the cause of the image offset in this scenario, thereby improving the image correction accuracy.

[0085] Optionally, in the image correction method provided by the embodiments of the present application, in the case where the image offset scenario is a scenario in which the image is offset due to weather factors or human factors, the target offset correction manner refers to a manner of performing offset correction based on an angle point matching algorithm, and the target offset correction manner is used to determine offset correction information based on the target offset correction manner, a target image, and a reference image, including: obtaining camera parameters of a target camera, wherein the camera parameters at least include distortion parameters; performing distortion correction on the reference image and the target image based on the camera parameters to obtain a first reference image and a first target image; matching angle points in the first reference image and the first target image based on an angle point detection algorithm to obtain a plurality of angle point pairs; determining a target homography matrix of the target image relative to the reference image based on the plurality of angle point pairs, and determining the offset correction information based on the target homography matrix.

[0086] Optionally, the camera parameters at least include distortion parameters, and the distortion parameters are used to correct lens distortion. The camera parameters can further include camera intrinsic parameters, and the camera intrinsic parameters at least include focal length (lateral focal length and longitudinal focal length) and optical center coordinates (lateral offset and longitudinal offset), wherein the focal length represents the projection length of the focal length of the camera lens on the image plane, and the optical center coordinates represent the position of the origin of the image coordinate system in the pixel coordinate system.

[0087] Optionally, the camera parameters of the target camera can be determined by manual pre-determination or by the target processing system based on images captured by the camera.

[0088] After obtaining the camera parameters of the target camera, the target processing system can perform distortion correction on the reference image and the target image based on the camera parameters to eliminate the distortion effect in the images, to obtain a first reference image and a first target image.

[0089] After obtaining the first reference image and the first target image, the target processing system can enhance high-frequency information in the first reference image by using a Laplacian operator to obtain a third reference image, and enhance high-frequency information in the first target image by using the Laplacian operator to obtain a second target image. Then, the third reference image and the second target image are each divided into a plurality of small image blocks, and angle points of the images are extracted in the small image blocks so that the distribution of the angle points is as uniform as possible, which can be achieved by using a related intersection detection algorithm. Then, the angle points in the third reference image and the second target image are matched by using a descriptor of the angle points to find corresponding angle point pairs in the two images, thereby obtaining a plurality of angle point pairs.

[0090] After the plurality of corner point pairs are obtained, a target homography matrix of the target image relative to the reference image can be determined based on the plurality of corner point pairs. For example, the target homography matrix can be determined based on a least square method and the plurality of corner point pairs, where the number of the plurality of corner point pairs is at least 4. For another example, the target homography matrix can be determined based on a RANSAC algorithm and the plurality of corner point pairs, such as converting a criterion of the RANSAC algorithm from the number of matched corner point pairs to the size of the region occupied by the matched corner point pairs, so as to eliminate the influence of the moving object occupying a small part of the scene, and thus obtaining the target homography matrix through the adjusted RANSAC algorithm. The RANSAC algorithm is an iterative algorithm for estimating the parameters of a mathematical model from a set of data containing outliers.

[0091] After the target homography matrix is obtained, the target homography matrix can be determined as the offset correction information.

[0092] It should be noted that, compared with the mutual information matching which is not sensitive to the change of illumination, the corner point matching algorithm is more suitable for the dynamic offset scene under the condition of fast changing or non-uniform illumination, because it can accurately identify the geometric features in the image, and can maintain high matching accuracy even under local illumination change, occlusion or view angle change, so as to improve the accuracy of determining the offset correction information.

[0093] Optionally, in the image correction method provided by the embodiments of the present application, the reference image includes annotation information, and the annotation information is used to identify a region of interest in the reference image. The image correction of the target image based on the offset correction information includes: mapping the annotation information in the reference image to the target image based on the target homography matrix to obtain an updated target image, so as to complete the image correction.

[0094] Optionally, the annotation information can be a rectangular frame or the like, and the annotation information can also be referred to as an electronic fence. The annotation information in the reference image is used to identify a region of interest in the reference image. For example, in the scene of monitoring the screen captured by the target camera, the region of interest in the reference image refers to the region that needs to be monitored or paid special attention to.

[0095] In the case of image offset caused by weather factors or human factors, since the picture captured by the target camera changes, and the annotation information is always in a fixed position in the captured picture, the region identified by the annotation information will not match the region that the user originally expected to identify. Moreover, in this case, the picture offset of the target camera caused by weather factors can be short-term, or adjusting the position of the annotation information will not affect the camera pose. Therefore, in this case, the annotation information in the reference image is mapped to the target image based on the target homography matrix to obtain an updated target image, so as to complete image correction.

[0096] For example, the annotation information in the reference image is mapped to the corresponding position of the target image by coordinate transformation based on the target homography matrix, so as to generate an updated target image.

[0097] In an optional embodiment, the target image itself also includes annotation information. In this case, the annotation information in the target image can be removed first, and then new annotation information is mapped based on the target homography matrix. The annotation information can be added to the reference image or the target image based on the preset annotation information coordinate information by the target processing system after the reference image or the target image is obtained. For example, relevant personnel can manually annotate or identify some position annotation or enclose an electronic fence on the monitoring picture (i.e., the reference image or the target image) by the target processing system when monitoring, so that the relevant personnel can better focus on the corresponding monitoring position or monitoring region when adjusting the camera pose or focal length of the high tower.

[0098] It should be noted that the above method can effectively offset the offset reason in the case of image offset caused by weather factors or human factors, thereby improving the accuracy of image correction.

[0099] Optionally, in the image correction method provided in the embodiments of the present application, the camera parameter of the target camera is obtained, including: determining a shooting scene of the target camera, wherein the shooting scene is one of the following: a scene in which an image obtained by shooting contains M three-dimensional straight lines, a scene in which an image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, and a scene in which an image obtained by shooting contains N three-dimensional straight lines and the target camera is not rotatable, M is greater than a preset value, and N is less than or equal to the preset value; in a case where the shooting scene is the scene in which the image obtained by shooting contains M three-dimensional straight lines, the camera parameter is determined based on a target Hough transform method; in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, the target camera is rotated, and the camera parameter is determined based on an image collected before the target camera is rotated and an image collected after the target camera is rotated; and in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is not rotatable, a plurality of three-dimensional target points are erected in a layout site of the target camera, and the camera parameter is determined based on a plurality of target point images obtained by the target camera shooting the plurality of three-dimensional target points.

[0100] Optionally, the shooting scene of the target camera can be determined by a user and input to the target processing system.

[0101] Optionally, in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, that is, if the scene does not contain enough three-dimensional straight lines (such as grassland, plain, etc.), the initial value of the distortion parameter is set to zero, a target camera fixed-axis rotation model is established, the target camera is rotated by a certain angle around a vertical axis, and two monitoring images (i.e., collected images) before and after the rotation are recorded respectively, distortion correction and camera intrinsic parameter estimation are performed based on the two images and camera depression angles and rotation angles when the two images are shot, and the estimated distortion parameter and camera intrinsic parameter are stored in a server.

[0102] For example, for a pixel coordinate [u, v] of a point on an image shot before rotation, which is corrected by a distortion parameter T , an intrinsic parameter matrix is K, a rotation matrix corresponding to a rotation motion is R, and a pixel coordinate [u', v'] of a corresponding point on an image after rotation, which is corrected by a distortion parameter T , a homography matrix corresponding to the two images is H, and a coordinate transformation in a rotation process can be illustrated by the following formula: Therefore, for the camera intrinsic parameters involved in the distortion parameters, the following loss can be defined: the image before rotation is mapped to the image coordinate system after rotation according to the above formula, and the pixel value error between the mapped image and the image after rotation is calculated as the target loss. The second objective function is designed to minimize the target loss, and the estimated camera parameters and distortion parameters are obtained based on the second objective function. In the process of obtaining the estimated camera parameters and distortion parameters based on the second objective function, the distortion parameters and the intrinsic matrix can be first set to the corresponding initial values, and then the distortion parameters and the intrinsic matrix are adjusted using an optimization algorithm to minimize the function value of the second objective function, so that the camera intrinsic parameters corresponding to the obtained distortion parameters and intrinsic matrix are determined as the camera parameters when the function value of the second objective function is less than a preset second threshold.

[0103] Optionally, in the case that the scene for shooting contains N three-dimensional straight lines in the obtained image and the target camera cannot rotate, that is, if the scene does not contain enough three-dimensional straight lines and the camera cannot rotate, a plurality of three-dimensional target points are erected in a scattered manner in the arrangement site of the target camera, and the target camera is used to shoot the plurality of three-dimensional target points to obtain a plurality of target point images, ensuring that part or all of the three-dimensional target points can be seen in each target point image, and recording the three-dimensional coordinates of the three-dimensional target points in the same coordinate system and the corresponding two-dimensional coordinates in the target point images. When determining the corresponding two-dimensional coordinates of the three-dimensional target points in the target point images, the target point images are first corrected based on the distortion parameters, and then the corresponding two-dimensional coordinates of the three-dimensional target points in the corrected target point images are determined. Then, a pinhole camera model is used to describe the camera imaging process, which assumes that light rays project through the optical center of the camera to the image plane. The intrinsic parameters and distortion parameters of the camera are optimized by minimizing the re-projection error under the pinhole camera model, and the optimization algorithm is used to adjust the intrinsic parameters and distortion parameters of the camera to minimize the re-projection error, so that the camera intrinsic parameters corresponding to the obtained distortion parameters and intrinsic matrix are determined as the camera parameters when the re-projection error is less than a preset third threshold.

[0104] Optionally, in the case that the scene for shooting contains M three-dimensional straight lines in the obtained image, that is, for scenes containing more three-dimensional straight lines (such as city centers, factories, etc.), an image is shot, and the two-dimensional projection of the three-dimensional straight line in the image is tried to be extracted, and the distortion parameters obtained by correcting the two-dimensional projection are obtained.

[0105] For example, edge information in the picture is extracted by an edge detection algorithm (e.g., Canny operator), and then distortion parameters are estimated based on a target Hough transform method (i.e., improved Hough transform). Specifically, first, the dimension of the distortion parameter is introduced in the parameter space of a straight line, the value range of the distortion parameter is divided into several discrete values, and a new three-dimensional Hough parameter space is formed. Then, for each discrete distortion parameter value, the edge points extracted in the previous step are de-distorted according to the value, and the tangent directions (i.e., the direction information of the edges) of the edge points are updated. Subsequently, each edge point votes for a straight line that is close to the edge point and has a direction close to the tangent direction of the edge point, and the closer the discrete point is to the straight line, the greater the voting value of the straight line. For a straight line, the sum of the voting values of all the discrete points is the voting value of the straight line. Finally, for each distortion parameter value, the P (P is a positive integer greater than 1) straight lines with the highest voting values under the parameter value are selected, and the voting values of the straight lines are added together. The maximum sum of the voting values corresponds to the distortion parameter of the camera parameter. Optionally, in the implementation process, the gimbal is controlled as much as possible to make the longer straight line at the edge of the image, so that it can be distorted to the greatest extent, so as to obtain more accurate distortion parameters.

[0106] Optionally, in the case where the shooting scene is a scene in which the obtained image contains M three-dimensional straight lines, after the distortion parameters are determined, the camera intrinsic parameters can be determined by using the processing manner of any one of the other two scenes (a scene in which the obtained image contains N three-dimensional straight lines and the target camera can rotate, and a scene in which the obtained image contains N three-dimensional straight lines and the target camera cannot rotate), and thus the details are not described herein. In the process of determining the camera intrinsic parameters, the parameter value of the distortion parameter determined in the scene in which the obtained image contains M three-dimensional straight lines can be used as an initial value to optimize the distortion parameter again.

[0107] It should be noted that through the above process, the camera parameters are accurately determined in different scenes, so that the accuracy of image correction can be improved.

[0108] As can be seen, the method provided in the present application achieves the purpose of correcting the image by using the offset correction manner corresponding to the image offset scene, and achieves the technical effect of improving the correction accuracy, and solves the technical problem of low correction accuracy when the camera pose changes.

[0109] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0110] Embodiment 2

[0111] The present application also provides an image correction device. It should be noted that the image correction device of the present application can be used to perform the image correction method provided in the present application. The image correction device provided in the present application is introduced below.

[0112] According to an embodiment of the present application, a device for implementing the above-mentioned image correction method is also provided, such as Figure 4 As shown, the device includes:

[0113] An acquisition module 401 is configured to acquire a target image acquired by a target camera at a target time and a reference image acquired at a reference time, wherein the target time is later than the reference time;

[0114] A first determining module 402 is configured to determine a target offset correction method for a target camera based on an image offset scenario between a target image and a reference image, wherein the image offset scenario is one of the following: a scenario in which image offset accumulates over time, or a scenario in which image offset is caused by weather factors or human factors;

[0115] A second determining module 403 is configured to determine offset correction information based on the target offset correction method, the target image, and the reference image;

[0116] The correction module 404 is configured to perform image correction on the target image based on the offset correction information.

[0117] In an embodiment of the present application, by acquiring a target image captured by the target camera at a target moment and a reference image captured at a reference moment, effective acquisition of images captured before and after the change in the target camera posture is achieved. By determining a target offset correction method for the target camera based on the image offset scenario between the target image and the reference image, a targeted determination of the offset correction method based on the scenario that causes the image offset between the target image and the reference image is achieved. Therefore, when determining the offset correction information based on the target offset correction method, the target image and the reference image, the offset correction information is targetedly determined based on the difference in image features captured before and after the change in the target camera posture and the specific correction method, which can effectively improve the correction accuracy during image correction.

[0118] It can be seen that the method provided in this application achieves the purpose of image correction using an offset correction method corresponding to the image offset scenario, realizes the technical effect of improving the correction accuracy, and solves the technical problem of low correction accuracy when the related technology corrects the image taken by the camera when the camera posture changes.

[0119] Optionally, in the image correction device provided in the embodiments of the present application, the image correction device further comprises: a third determination module, configured to determine, in a case where a scene triggering instruction of the user is received, that the image offset scene is a scene in which the image offset is caused by weather factors or human factors; and a fourth determination module, configured to determine, in a case where no scene triggering instruction is received and the target moment is a preset moment, that the image offset scene is a scene in which the image offset is caused by time accumulation.

[0120] Optionally, in the image correction device provided in the embodiments of the present application, the second determination module further comprises: a first acquisition sub-module, configured to acquire camera parameters of the target camera, wherein the camera parameters at least include distortion parameters; a first correction sub-module, configured to perform distortion correction on the reference image and the target image respectively based on the camera parameters, to obtain a first reference image and a first target image; a first mapping sub-module, configured to construct a homography matrix based on the initial offset parameters, and perform image mapping on the first reference image based on the homography matrix, to obtain a second reference image; and a first determination sub-module, configured to determine the offset parameters of the target image relative to the reference image based on a first target function and mutual information between the second reference image and the first target image, and determine the offset correction information based on the offset parameters, wherein the first target function takes minimizing the reciprocal of the mutual information as a target.

[0121] Optionally, in the image correction device provided in the embodiments of the present application, the correction module further comprises: an adjustment sub-module, configured to adjust the pose of the target camera based on the offset parameters; and an acquisition sub-module, configured to perform image acquisition based on the target camera after the pose adjustment, to obtain an updated target image, so as to complete the image correction.

[0122] Optionally, in the image correction device provided in the embodiments of the present application, the second determination module further comprises: a second acquisition sub-module, configured to acquire camera parameters of the target camera, wherein the camera parameters at least include distortion parameters; a second correction sub-module, configured to perform distortion correction on the reference image and the target image respectively based on the camera parameters, to obtain a first reference image and a first target image; a matching sub-module, configured to match the corner points in the first reference image and the first target image based on a corner detection algorithm, to obtain a plurality of corner point pairs; and a second determination sub-module, configured to determine a target homography matrix of the target image relative to the reference image based on the plurality of corner point pairs, and determine the offset correction information based on the target homography matrix.

[0123] Optionally, in the image correction device provided in the embodiments of the present application, the correction module further comprises: a second mapping sub-module, configured to map the annotation information in the reference image into the target image based on the target homography matrix, to obtain an updated target image, so as to complete the image correction.

[0124] Optionally, in the image correction device provided in the embodiment of the present application, the first acquisition submodule or the second acquisition submodule further includes: a third determination submodule, configured to determine a shooting scene of the target camera, wherein the shooting scene is one of the following: a scene in which the captured image contains M three-dimensional straight lines, a scene in which the captured image contains N three-dimensional straight lines and the target camera is rotatable, or a scene in which the captured image contains N three-dimensional straight lines and the target camera cannot be rotated, where M is greater than a preset value and N is less than or equal to a preset value; a fourth determination submodule, configured to determine camera parameters based on a target Hough transform method when the shooting scene is a scene in which the captured image contains M three-dimensional straight lines; a fifth determination submodule, configured to rotate the target camera when the shooting scene is a scene in which the captured image contains N three-dimensional straight lines and the target camera is rotatable, and determine the camera parameters based on the captured image before and after the rotation of the target camera; and a sixth determination submodule, configured to set up multiple three-dimensional targets in the target camera arrangement area when the shooting scene is a scene in which the captured image contains N three-dimensional straight lines and the target camera cannot be rotated, and determine the camera parameters based on multiple target point images obtained by capturing the multiple three-dimensional targets by the target camera.

[0125] It should be noted that the acquisition module 401, the first determination module 402, the second determination module 403, and the correction module 404 correspond to steps S201 to S204 in Example 1. The four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above-mentioned modules can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0126] Example 3

[0127] An embodiment of the present application may provide an electronic device, Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 Only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0128] The memory can be configured to store software programs and modules, such as program instructions / modules corresponding to the method and device in the embodiments of the present application. The processor executes various functions and data processing by running the software programs and modules stored in the memory, that is, implements the above method. The memory can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0129] The processor can call information and applications stored in the memory through the transmission device to perform the following steps: obtaining a target image collected by a target camera at a target time and a reference image collected at a reference time, wherein the target time is later than the reference time; determining a target offset correction mode for the target camera according to an image offset scene between the target image and the reference image, wherein the image offset scene is one of the following: a scene in which image offset is accumulated over time, a scene in which image offset is caused by weather factors or human factors; determining offset correction information based on the target offset correction mode, the target image, and the reference image; and performing image correction on the target image based on the offset correction information.

[0130] The processor can also call information and applications stored in the memory through the transmission device to perform the following steps: in the case that the scene trigger instruction of the user is received, determining that the image offset scene is the scene in which the image offset is caused by weather factors or human factors, before determining the target offset correction mode for the target camera according to the image offset scene between the target image and the reference image; in the case that the scene trigger instruction is not received and the target time is a preset time, determining that the image offset scene is the scene in which the image offset is accumulated over time.

[0131] The processor can also call information and applications stored in the memory through the transmission device to perform the following steps: obtaining camera parameters of the target camera, wherein the camera parameters at least include distortion parameters; performing distortion correction on the reference image and the target image based on the camera parameters to obtain a first reference image and a first target image; constructing a homography matrix based on an initial offset parameter, and performing image mapping on the first reference image based on the homography matrix to obtain a second reference image; determining an offset parameter of the target image relative to the reference image based on a first target function and mutual information between the second reference image and the first target image, and determining offset correction information based on the offset parameter, wherein the first target function aims to minimize the reciprocal of the mutual information.

[0132] The processor can further call information and application programs stored in the memory through the transmission device to perform the following steps: adjusting the pose of the target camera based on the offset parameter; and performing image acquisition based on the target camera after the pose is adjusted to obtain an updated target image, so as to complete image correction.

[0133] The processor can further call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining camera parameters of the target camera, wherein the camera parameters at least include distortion parameters; performing distortion correction on the reference image and the target image based on the camera parameters to obtain a first reference image and a first target image; matching the corner points in the first reference image and the first target image based on a corner detection algorithm to obtain a plurality of corner point pairs; determining a target homography matrix of the target image relative to the reference image based on the plurality of corner point pairs, and determining offset correction information based on the target homography matrix.

[0134] The processor can further call information and application programs stored in the memory through the transmission device to perform the following steps: mapping the annotation information in the reference image into the target image based on the target homography matrix to obtain an updated target image, so as to complete image correction.

[0135] The processor can further call information and application programs stored in the memory through the transmission device to perform the following steps: determining a shooting scene of the target camera, wherein the shooting scene is one of the following: a scene in which the image obtained by shooting contains M three-dimensional straight lines, a scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, and a scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is not rotatable, M is greater than a preset value, and N is less than or equal to the preset value; in a case where the shooting scene is the scene in which the image obtained by shooting contains M three-dimensional straight lines, determining the camera parameters based on a target Hough transform method; in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, rotating the target camera, and determining the camera parameters based on the acquisition image before the target camera is rotated and the acquisition image after the target camera is rotated; in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is not rotatable, erecting a plurality of three-dimensional target points in the arrangement site of the target camera, and determining the camera parameters based on a plurality of target point images obtained by the target camera shooting the plurality of three-dimensional target points.

[0136] In the embodiment of the present application, the target image collected by the target camera at the target time and the reference image collected at the reference time are acquired, the image collected before and after the change of the target camera pose is effectively acquired, the target offset correction mode for the target camera is determined according to the image offset scene between the target image and the reference image, the offset correction mode is determined based on the scene causing the image offset between the target image and the reference image, and thus when the offset correction information is determined based on the target offset correction mode, the target image and the reference image, the offset correction information is determined based on the difference between the images collected before and after the change of the target camera pose and the specific correction mode, and the correction accuracy during image correction can be effectively improved.

[0137] Therefore, the method provided in the present application achieves the purpose of image correction by using the offset correction mode corresponding to the image offset scene, achieves the technical effect of improving the correction accuracy, and solves the technical problem of low correction accuracy when the image captured by the camera is corrected in the case of the change of the camera pose.

[0138] Those skilled in the art can understand that Figure 5 The structure shown is only schematic, and the electronic device can also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, etc. Figure 5 It does not limit the structure of the electronic device. For example, the electronic device can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 5 It does not limit the structure of the electronic device. For example, the electronic device can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 5 It does not limit the structure of the electronic device. For example, the electronic device can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure.

[0139] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing the related hardware of the terminal device, and the programs can be stored in a computer readable storage medium, which can include a flash disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0140] Embodiment 4

[0141] The embodiment of the present application also provides a storage medium. Optionally, in the embodiment, the storage medium can be used to save the program code executed by the image correction method provided in Embodiment 1.

[0142] Optionally, in the embodiment, the storage medium can be located in any one of computer terminals in a computer terminal group in a computer network, or in any one of mobile terminals in a mobile terminal group.

[0143] The application also provides a computer program product, which is suitable for executing the steps of the image correction method when executed on a data processing device.

[0144] The serial numbers of the embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0145] In the above-described embodiments of the application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0146] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only a schematic and illustrative, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interface, and can be electrical or other forms.

[0147] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0148] In addition, each functional unit in each embodiment of the application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0149] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0150] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. An image correction method characterized by, The method comprises the following steps: acquiring a target image collected by a target camera at a target time and a reference image collected at a reference time, wherein the target time is later than the reference time; determining a target offset correction mode for the target camera according to an image offset scenario between the target image and the reference image, wherein the image offset scenario is one of the following scenarios: a scenario in which image offset is accumulated over time, and a scenario in which image offset is caused by weather factors or human factors; determining offset correction information based on the target offset correction mode, the target image, and the reference image; performing image correction on the target image based on the offset correction information.

2. The method of claim 1, wherein, Before determining the target offset correction mode for the target camera according to the image offset scenario between the target image and the reference image, the method further comprises the following steps: in the case where a scenario triggering instruction of a user is received, determining that the image offset scenario is the scenario in which image offset is caused by weather factors or human factors; in the case where the scenario triggering instruction is not received and the target time is a preset time, determining that the image offset scenario is the scenario in which image offset is accumulated over time.

3. The method of claim 1, wherein, In the case where the image offset scenario is the scenario in which image offset is accumulated over time, the target offset correction mode refers to a mode of performing offset correction based on mutual information, wherein the determination of the offset correction information based on the target offset correction mode, the target image, and the reference image comprises the following steps: acquiring camera parameters of the target camera, wherein the camera parameters at least include distortion parameters; performing distortion correction on the reference image and the target image based on the camera parameters to obtain a first reference image and a first target image; constructing a homography matrix based on an initial offset parameter, and performing image mapping on the first reference image based on the homography matrix to obtain a second reference image; determining an offset parameter of the target image relative to the reference image based on a first target function and mutual information between the second reference image and the first target image, and determining the offset correction information based on the offset parameter, wherein the first target function aims to minimize the reciprocal of the mutual information.

4. The method of claim 3, wherein, The image correction on the target image based on the offset correction information comprises the following steps: adjusting a pose of the target camera based on the offset parameter; performing image collection based on the target camera after the pose adjustment to obtain an updated target image, so as to complete the image correction.

5. The method of claim 1, wherein, In the case where the image offset scenario is the scenario in which image offset is caused by weather factors or human factors, the target offset correction mode refers to a mode of performing offset correction based on a corner point matching algorithm, wherein the determination of the offset correction information based on the target offset correction mode, the target image, and the reference image comprises the following steps: acquiring camera parameters of the target camera, wherein the camera parameters at least include distortion parameters; performing distortion correction on the reference image and the target image based on the camera parameters to obtain a first reference image and a first target image; match the corner points in the first reference image and the first target image based on a corner point detection algorithm to obtain a plurality of corner point pairs; determine a target homography matrix of the target image relative to the reference image based on the plurality of corner point pairs, and determine the offset correction information based on the target homography matrix.

6. The method of claim 5, wherein, The reference image includes annotation information, and the annotation information is used to identify a region of interest in the reference image. The image correction of the target image based on the offset correction information includes: mapping the annotation information in the reference image into the target image based on the target homography matrix to obtain an updated target image, so as to complete the image correction.

7. The method according to claim 3 or 5, characterized in that, obtaining camera parameters of the target camera, including: determining a shooting scene of the target camera, wherein the shooting scene is one of the following: a scene in which an image obtained by shooting contains M three-dimensional straight lines, a scene in which an image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, and a scene in which an image obtained by shooting contains N three-dimensional straight lines and the target camera is not rotatable, M is greater than a preset value, and N is less than or equal to the preset value; in a case where the shooting scene is the scene in which the image obtained by shooting contains M three-dimensional straight lines, determining the camera parameters based on a target Hough transform method; in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is rotatable, rotating the target camera, and determining the camera parameters based on an image collected before the target camera is rotated and an image collected after the target camera is rotated; in a case where the shooting scene is the scene in which the image obtained by shooting contains N three-dimensional straight lines and the target camera is not rotatable, erecting a plurality of three-dimensional target points in a layout site of the target camera, and determining the camera parameters based on a plurality of target point images obtained by the target camera shooting the plurality of three-dimensional target points.

8. An image correction apparatus characterized by comprising: including: an obtaining module configured to obtain a target image collected by a target camera at a target time and a reference image collected by the target camera at a reference time, wherein the target time is later than the reference time; a first determining module configured to determine a target offset correction mode for the target camera according to an image offset scenario between the target image and the reference image, wherein the image offset scenario is one of the following: a scenario in which image offset is caused by time accumulation, and a scenario in which image offset is caused by weather factors or human factors; a second determining module configured to determine offset correction information based on the target offset correction mode, the target image, and the reference image; a correction module configured to perform image correction on the target image based on the offset correction information.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein the executable program controls a device in which the computer-readable storage medium is located to perform the image correction method of any one of claims 1 to 7 when the executable program is running.

10. An electronic device, comprising: including: a memory storing an executable program; a processor configured to run the program, wherein the program performs the image correction method of any one of claims 1 to 7 when the program is running.

Images